Lazily Enabled Truth Maintenace (Project Propsoal)

classic Classic list List threaded Threaded
17 messages Options
Reply | Threaded
Open this post in threaded view
|

Lazily Enabled Truth Maintenace (Project Propsoal)

Mark Proctor
Here is another project proposal, this time simpler. I think this one has Wolfgang's name on it ;)

http://blog.athico.com/2010/09/lazily-enabled-truth-maintenace.html

Three weeks ago I posted the project idea for "Left and Right Unlinking". So far there are no takers, so if you are interested let me know :)

In the meantime I tried to think of a simpler enhancement that we would like to see done.

At the moment Drools has a user setting "MaintainTMSOption" which can be true or false. It's a small optimisation that when turned off avoids using the equality hashmap that is maintained for all inserted objects.

It would be a much better idea to remove this configuration setting, thus simplifying things for end users and have TMS lazily enabled on demand.

For each object type there is an "ObjectTypeConf" configuration object that is retrieved every time a working memory action, such as insert, is executed. The enabledTMS boolean should be moved there, so there is one per object type, by default it is false.

When a working memory action occurs, like insert, it retrieved the ObjectTypeConf and checks the maintainTms boolean there, instead of the current engine scoped configuration. When a logical insertion occurs and the ObjectTypeConf is retrieved if maintainTms is false it sets the value to true and then iterates the associated ObjectTypeNode memory lazily adding all the objects to the TMS equality map. From then on for that ObjectType all inserted objects are added to that equality map.

With this you now have the advantage of TMS being laziy enabled, so the minor hashmap operation is no longer used and likewise a small memory saving from not populating the map. There is a further advantage that this is now fine grained and when enabled only impacts for that specific object type.

A further enhancement could use a int counter, instead of a boolean. Each logical insertion for that object type increases the counter, each retraction decreases the counter; even if automatically retracted if the truth is broken for that logical assertion. When the counter reaches zero, TMS for that OTN can be disabled. We do not however remove the objects from the equality map, as this would cause "churn" if TMS is continuously enabled and disabled. Instead when TMS is disabled record the current fact counter id. Then if TMS is disabled on a retraction but there is a counter id, we can check that counter id to see if the fact is prior to TMS being disabled and thus would need to be retracted from the equality map.

_______________________________________________
rules-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/rules-dev
Reply | Threaded
Open this post in threaded view
|

Re: Lazily Enabled Truth Maintenace (Project Propsoal)

manstis
Is this in drools-core; or drools-compiler?

Whilst not undertaking to do the work; have a purpose to nose through the code makes understanding easier.

On 14 September 2010 16:47, Mark Proctor <[hidden email]> wrote:
Here is another project proposal, this time simpler. I think this one has Wolfgang's name on it ;)

http://blog.athico.com/2010/09/lazily-enabled-truth-maintenace.html

Three weeks ago I posted the project idea for "Left and Right Unlinking". So far there are no takers, so if you are interested let me know :)

In the meantime I tried to think of a simpler enhancement that we would like to see done.

At the moment Drools has a user setting "MaintainTMSOption" which can be true or false. It's a small optimisation that when turned off avoids using the equality hashmap that is maintained for all inserted objects.

It would be a much better idea to remove this configuration setting, thus simplifying things for end users and have TMS lazily enabled on demand.

For each object type there is an "ObjectTypeConf" configuration object that is retrieved every time a working memory action, such as insert, is executed. The enabledTMS boolean should be moved there, so there is one per object type, by default it is false.

When a working memory action occurs, like insert, it retrieved the ObjectTypeConf and checks the maintainTms boolean there, instead of the current engine scoped configuration. When a logical insertion occurs and the ObjectTypeConf is retrieved if maintainTms is false it sets the value to true and then iterates the associated ObjectTypeNode memory lazily adding all the objects to the TMS equality map. From then on for that ObjectType all inserted objects are added to that equality map.

With this you now have the advantage of TMS being laziy enabled, so the minor hashmap operation is no longer used and likewise a small memory saving from not populating the map. There is a further advantage that this is now fine grained and when enabled only impacts for that specific object type.

A further enhancement could use a int counter, instead of a boolean. Each logical insertion for that object type increases the counter, each retraction decreases the counter; even if automatically retracted if the truth is broken for that logical assertion. When the counter reaches zero, TMS for that OTN can be disabled. We do not however remove the objects from the equality map, as this would cause "churn" if TMS is continuously enabled and disabled. Instead when TMS is disabled record the current fact counter id. Then if TMS is disabled on a retraction but there is a counter id, we can check that counter id to see if the fact is prior to TMS being disabled and thus would need to be retracted from the equality map.

_______________________________________________
rules-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/rules-dev



_______________________________________________
rules-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/rules-dev
Reply | Threaded
Open this post in threaded view
|

Re: Lazily Enabled Truth Maintenace (Project Propsoal)

Mark Proctor
On 15/09/2010 14:35, Michael Anstis wrote:
Is this in drools-core; or drools-compiler?

Whilst not undertaking to do the work; have a purpose to nose through the code makes understanding easier.
It's all in DroolsCore.

It's a 5 minute hack for me and then 15 minute unit writing test. But I thought I'd write it up in a hope to bring someone else into the fold, we need more help writting the core engine someone else out there must want to work on current edge engine design :)

if you look at the AbstractWorkingMemory insert method you'll see one argument is whether it's a logical insertion or not. You'll also see it check the global maintainTMS configuration and also retrieve the ObjectTypeConf. So between those things someone should be able to get it working.

Mark

On 14 September 2010 16:47, Mark Proctor <[hidden email]> wrote:
Here is another project proposal, this time simpler. I think this one has Wolfgang's name on it ;)

http://blog.athico.com/2010/09/lazily-enabled-truth-maintenace.html

Three weeks ago I posted the project idea for "Left and Right Unlinking". So far there are no takers, so if you are interested let me know :)

In the meantime I tried to think of a simpler enhancement that we would like to see done.

At the moment Drools has a user setting "MaintainTMSOption" which can be true or false. It's a small optimisation that when turned off avoids using the equality hashmap that is maintained for all inserted objects.

It would be a much better idea to remove this configuration setting, thus simplifying things for end users and have TMS lazily enabled on demand.

For each object type there is an "ObjectTypeConf" configuration object that is retrieved every time a working memory action, such as insert, is executed. The enabledTMS boolean should be moved there, so there is one per object type, by default it is false.

When a working memory action occurs, like insert, it retrieved the ObjectTypeConf and checks the maintainTms boolean there, instead of the current engine scoped configuration. When a logical insertion occurs and the ObjectTypeConf is retrieved if maintainTms is false it sets the value to true and then iterates the associated ObjectTypeNode memory lazily adding all the objects to the TMS equality map. From then on for that ObjectType all inserted objects are added to that equality map.

With this you now have the advantage of TMS being laziy enabled, so the minor hashmap operation is no longer used and likewise a small memory saving from not populating the map. There is a further advantage that this is now fine grained and when enabled only impacts for that specific object type.

A further enhancement could use a int counter, instead of a boolean. Each logical insertion for that object type increases the counter, each retraction decreases the counter; even if automatically retracted if the truth is broken for that logical assertion. When the counter reaches zero, TMS for that OTN can be disabled. We do not however remove the objects from the equality map, as this would cause "churn" if TMS is continuously enabled and disabled. Instead when TMS is disabled record the current fact counter id. Then if TMS is disabled on a retraction but there is a counter id, we can check that counter id to see if the fact is prior to TMS being disabled and thus would need to be retracted from the equality map.

_______________________________________________
rules-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/rules-dev


_______________________________________________ rules-dev mailing list [hidden email] https://lists.jboss.org/mailman/listinfo/rules-dev


_______________________________________________
rules-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/rules-dev
Reply | Threaded
Open this post in threaded view
|

Re: Lazily Enabled Truth Maintenace (Project Propsoal)

manstis
I had a go at doing this last night, but some unit tests now fail and two error; so something obviously isn't right.

I'll get back to it this evening and try to work out where I've gone wrong (before asking for guidance to help me understand better).

On 15 September 2010 14:47, Mark Proctor <[hidden email]> wrote:
On 15/09/2010 14:35, Michael Anstis wrote:
Is this in drools-core; or drools-compiler?

Whilst not undertaking to do the work; have a purpose to nose through the code makes understanding easier.
It's all in DroolsCore.

It's a 5 minute hack for me and then 15 minute unit writing test. But I thought I'd write it up in a hope to bring someone else into the fold, we need more help writting the core engine someone else out there must want to work on current edge engine design :)

if you look at the AbstractWorkingMemory insert method you'll see one argument is whether it's a logical insertion or not. You'll also see it check the global maintainTMS configuration and also retrieve the ObjectTypeConf. So between those things someone should be able to get it working.

Mark


On 14 September 2010 16:47, Mark Proctor <[hidden email]> wrote:
Here is another project proposal, this time simpler. I think this one has Wolfgang's name on it ;)

http://blog.athico.com/2010/09/lazily-enabled-truth-maintenace.html

Three weeks ago I posted the project idea for "Left and Right Unlinking". So far there are no takers, so if you are interested let me know :)

In the meantime I tried to think of a simpler enhancement that we would like to see done.

At the moment Drools has a user setting "MaintainTMSOption" which can be true or false. It's a small optimisation that when turned off avoids using the equality hashmap that is maintained for all inserted objects.

It would be a much better idea to remove this configuration setting, thus simplifying things for end users and have TMS lazily enabled on demand.

For each object type there is an "ObjectTypeConf" configuration object that is retrieved every time a working memory action, such as insert, is executed. The enabledTMS boolean should be moved there, so there is one per object type, by default it is false.

When a working memory action occurs, like insert, it retrieved the ObjectTypeConf and checks the maintainTms boolean there, instead of the current engine scoped configuration. When a logical insertion occurs and the ObjectTypeConf is retrieved if maintainTms is false it sets the value to true and then iterates the associated ObjectTypeNode memory lazily adding all the objects to the TMS equality map. From then on for that ObjectType all inserted objects are added to that equality map.

With this you now have the advantage of TMS being laziy enabled, so the minor hashmap operation is no longer used and likewise a small memory saving from not populating the map. There is a further advantage that this is now fine grained and when enabled only impacts for that specific object type.

A further enhancement could use a int counter, instead of a boolean. Each logical insertion for that object type increases the counter, each retraction decreases the counter; even if automatically retracted if the truth is broken for that logical assertion. When the counter reaches zero, TMS for that OTN can be disabled. We do not however remove the objects from the equality map, as this would cause "churn" if TMS is continuously enabled and disabled. Instead when TMS is disabled record the current fact counter id. Then if TMS is disabled on a retraction but there is a counter id, we can check that counter id to see if the fact is prior to TMS being disabled and thus would need to be retracted from the equality map.

_______________________________________________
rules-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/rules-dev


_______________________________________________ rules-dev mailing list [hidden email] https://lists.jboss.org/mailman/listinfo/rules-dev


_______________________________________________
rules-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/rules-dev



_______________________________________________
rules-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/rules-dev
Reply | Threaded
Open this post in threaded view
|

Re: Lazily Enabled Truth Maintenace (Project Propsoal)

leo_gomes
In reply to this post by Mark Proctor
if you look at the AbstractWorkingMemory insert method you'll see one argument is whether it's a logical insertion or not. You'll also see it check the global maintainTMS configuration and also retrieve the ObjectTypeConf. So between those things someone should be able to get it working.

Today, it enters a block where it operates on the equality map and also creates a default handle based on that TMS global option and *regardless* of whether it's a logical insert.

If I'm *not* putting things in the equality map for regular inserts, when a logical insert comes in, but there were already stated inserts, how will I know that? I would create a new handle for the logical insert and do the tms.addLogicalDependency(...), even tough there were regular inserts before and this seems to be a wrong behaviour.

Apparently, today, you can disable TMS and still use logical inserts in your drl, what, I believe, will lead to inconsistent behaviour, but you're at your own risk.

Ideas? I feel that I missed something :)

Cheers,
Leo.

P.S.: I reached the conclusions above based on the fact that LogicalAssertionTest started failing after I did the changes you suggested.

 


On Wed, Sep 15, 2010 at 3:47 PM, Mark Proctor <[hidden email]> wrote:
On 15/09/2010 14:35, Michael Anstis wrote:
Is this in drools-core; or drools-compiler?

Whilst not undertaking to do the work; have a purpose to nose through the code makes understanding easier.
It's all in DroolsCore.

It's a 5 minute hack for me and then 15 minute unit writing test. But I thought I'd write it up in a hope to bring someone else into the fold, we need more help writting the core engine someone else out there must want to work on current edge engine design :)

if you look at the AbstractWorkingMemory insert method you'll see one argument is whether it's a logical insertion or not. You'll also see it check the global maintainTMS configuration and also retrieve the ObjectTypeConf. So between those things someone should be able to get it working.

Mark


On 14 September 2010 16:47, Mark Proctor <[hidden email]> wrote:
Here is another project proposal, this time simpler. I think this one has Wolfgang's name on it ;)

http://blog.athico.com/2010/09/lazily-enabled-truth-maintenace.html

Three weeks ago I posted the project idea for "Left and Right Unlinking". So far there are no takers, so if you are interested let me know :)

In the meantime I tried to think of a simpler enhancement that we would like to see done.

At the moment Drools has a user setting "MaintainTMSOption" which can be true or false. It's a small optimisation that when turned off avoids using the equality hashmap that is maintained for all inserted objects.

It would be a much better idea to remove this configuration setting, thus simplifying things for end users and have TMS lazily enabled on demand.

For each object type there is an "ObjectTypeConf" configuration object that is retrieved every time a working memory action, such as insert, is executed. The enabledTMS boolean should be moved there, so there is one per object type, by default it is false.

When a working memory action occurs, like insert, it retrieved the ObjectTypeConf and checks the maintainTms boolean there, instead of the current engine scoped configuration. When a logical insertion occurs and the ObjectTypeConf is retrieved if maintainTms is false it sets the value to true and then iterates the associated ObjectTypeNode memory lazily adding all the objects to the TMS equality map. From then on for that ObjectType all inserted objects are added to that equality map.

With this you now have the advantage of TMS being laziy enabled, so the minor hashmap operation is no longer used and likewise a small memory saving from not populating the map. There is a further advantage that this is now fine grained and when enabled only impacts for that specific object type.

A further enhancement could use a int counter, instead of a boolean. Each logical insertion for that object type increases the counter, each retraction decreases the counter; even if automatically retracted if the truth is broken for that logical assertion. When the counter reaches zero, TMS for that OTN can be disabled. We do not however remove the objects from the equality map, as this would cause "churn" if TMS is continuously enabled and disabled. Instead when TMS is disabled record the current fact counter id. Then if TMS is disabled on a retraction but there is a counter id, we can check that counter id to see if the fact is prior to TMS being disabled and thus would need to be retracted from the equality map.

_______________________________________________
rules-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/rules-dev


_______________________________________________ rules-dev mailing list [hidden email] https://lists.jboss.org/mailman/listinfo/rules-dev


_______________________________________________
rules-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/rules-dev



_______________________________________________
rules-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/rules-dev
Reply | Threaded
Open this post in threaded view
|

Re: Lazily Enabled Truth Maintenace (Project Propsoal)

leo_gomes
I think I wasn't quite clear in my last email, so let me try to reformulate it:

I also gave it a try to try to do what you suggested here (http://blog.athico.com/2010/09/lazily-enabled-truth-maintenace.html), Mark, and couldn't  make it work due to the following situation:

1) Current code seems to rely on the equality map to know that a logical insert (insertLogical) for an object that has already been regularly inserted (insert) should be ignored;

2) If I apply the modifications that you suggested, from what I understood, things would start to be put in the equality map only when a logical insert is issued;

*Problem*: How would I handle the situation described in item 1, if I don't have anything in the equality map at the moment a logical insert comes in and I have to "lazily activate" TMS?

--

Moreover, I'm willing to attempt to implement the left and right un-linking and tried to start with this easier task to start to get familiar with drools-core. I already read the article you linked and your article. Would you have any document with an overview of the way drools implements rete?

Thanks in advance!

Cheers,
Leo.






On Sat, Sep 18, 2010 at 3:49 AM, Leonardo Gomes <[hidden email]> wrote:
if you look at the AbstractWorkingMemory insert method you'll see one argument is whether it's a logical insertion or not. You'll also see it check the global maintainTMS configuration and also retrieve the ObjectTypeConf. So between those things someone should be able to get it working.

Today, it enters a block where it operates on the equality map and also creates a default handle based on that TMS global option and *regardless* of whether it's a logical insert.

If I'm *not* putting things in the equality map for regular inserts, when a logical insert comes in, but there were already stated inserts, how will I know that? I would create a new handle for the logical insert and do the tms.addLogicalDependency(...), even tough there were regular inserts before and this seems to be a wrong behaviour.

Apparently, today, you can disable TMS and still use logical inserts in your drl, what, I believe, will lead to inconsistent behaviour, but you're at your own risk.

Ideas? I feel that I missed something :)

Cheers,
Leo.

P.S.: I reached the conclusions above based on the fact that LogicalAssertionTest started failing after I did the changes you suggested.

 


On Wed, Sep 15, 2010 at 3:47 PM, Mark Proctor <[hidden email]> wrote:
On 15/09/2010 14:35, Michael Anstis wrote:
Is this in drools-core; or drools-compiler?

Whilst not undertaking to do the work; have a purpose to nose through the code makes understanding easier.
It's all in DroolsCore.

It's a 5 minute hack for me and then 15 minute unit writing test. But I thought I'd write it up in a hope to bring someone else into the fold, we need more help writting the core engine someone else out there must want to work on current edge engine design :)

if you look at the AbstractWorkingMemory insert method you'll see one argument is whether it's a logical insertion or not. You'll also see it check the global maintainTMS configuration and also retrieve the ObjectTypeConf. So between those things someone should be able to get it working.

Mark


On 14 September 2010 16:47, Mark Proctor <[hidden email]> wrote:
Here is another project proposal, this time simpler. I think this one has Wolfgang's name on it ;)

http://blog.athico.com/2010/09/lazily-enabled-truth-maintenace.html

Three weeks ago I posted the project idea for "Left and Right Unlinking". So far there are no takers, so if you are interested let me know :)

In the meantime I tried to think of a simpler enhancement that we would like to see done.

At the moment Drools has a user setting "MaintainTMSOption" which can be true or false. It's a small optimisation that when turned off avoids using the equality hashmap that is maintained for all inserted objects.

It would be a much better idea to remove this configuration setting, thus simplifying things for end users and have TMS lazily enabled on demand.

For each object type there is an "ObjectTypeConf" configuration object that is retrieved every time a working memory action, such as insert, is executed. The enabledTMS boolean should be moved there, so there is one per object type, by default it is false.

When a working memory action occurs, like insert, it retrieved the ObjectTypeConf and checks the maintainTms boolean there, instead of the current engine scoped configuration. When a logical insertion occurs and the ObjectTypeConf is retrieved if maintainTms is false it sets the value to true and then iterates the associated ObjectTypeNode memory lazily adding all the objects to the TMS equality map. From then on for that ObjectType all inserted objects are added to that equality map.

With this you now have the advantage of TMS being laziy enabled, so the minor hashmap operation is no longer used and likewise a small memory saving from not populating the map. There is a further advantage that this is now fine grained and when enabled only impacts for that specific object type.

A further enhancement could use a int counter, instead of a boolean. Each logical insertion for that object type increases the counter, each retraction decreases the counter; even if automatically retracted if the truth is broken for that logical assertion. When the counter reaches zero, TMS for that OTN can be disabled. We do not however remove the objects from the equality map, as this would cause "churn" if TMS is continuously enabled and disabled. Instead when TMS is disabled record the current fact counter id. Then if TMS is disabled on a retraction but there is a counter id, we can check that counter id to see if the fact is prior to TMS being disabled and thus would need to be retracted from the equality map.

_______________________________________________
rules-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/rules-dev


_______________________________________________ rules-dev mailing list [hidden email] https://lists.jboss.org/mailman/listinfo/rules-dev


_______________________________________________
rules-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/rules-dev




_______________________________________________
rules-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/rules-dev
Reply | Threaded
Open this post in threaded view
|

Re: Lazily Enabled Truth Maintenace (Project Propsoal)

Mark Proctor
On 20/09/2010 15:53, Leonardo Gomes wrote:
I think I wasn't quite clear in my last email, so let me try to reformulate it:

I also gave it a try to try to do what you suggested here (http://blog.athico.com/2010/09/lazily-enabled-truth-maintenace.html), Mark, and couldn't  make it work due to the following situation:

1) Current code seems to rely on the equality map to know that a logical insert (insertLogical) for an object that has already been regularly inserted (insert) should be ignored;
You would need to lazily maintain an equality map. When the first logical insertion is done we will have to first populate that map from the Object Type Node set of FactHandles.
In the "insert" one of the first lines is:
            ObjectTypeConf typeConf = this.typeConfReg.getObjectTypeConf( this.entryPoint,
                                                                          object );

So we get the ObjectTypeConf before we do anything with the object itself, and we can check if TMS is being maintained for that Object Type.

2) If I apply the modifications that you suggested, from what I understood, things would start to be put in the equality map only when a logical insert is issued;

*Problem*: How would I handle the situation described in item 1, if I don't have anything in the equality map at the moment a logical insert comes in and I have to "lazily activate" TMS?
final Rete source = this.ruleBase.getRete();
ClassObjectType cot = new ClassObjectType( MyClass.class );
Map<ObjectType, ObjectTypeNode> map = source.getObjectTypeNodes( EntryPoint.DEFAULT );
ObjectTypeNode node = map.get( cot );
final ObjectHashSet memory = (ObjectHashSet) workingMemory.getNodeMemory( node );

That "memory" is the set of currently asserted objects for that OTN. You can now iterate that and populate the equality hash map. This is a one off as the flag will be set on the ObjectTypeConf from then on and all objects will be added to the equality map at the point of insertion.

Does that help? All the information is there, you should have to know how to retrieve it :) We don't currently maintain TMS for anything other than the default entry point. Although I think that was a mistake and we will probably move all entrypoints to work with same, but for now you can ignore that and just focus on the default.

Btw this should not be confused with "equality" mode where the equality map has to be maintained by default from the start. Some would argue that a rule engine should only under work and understand on the bases of equality and thus all users must implement hashcode and equals correctly...

--

Moreover, I'm willing to attempt to implement the left and right un-linking and tried to start with this easier task to start to get familiar with drools-core. I already read the article you linked and your article. Would you have any document with an overview of the way drools implements rete?
not really no, Rete has already well documented in a number of papers. The best thing to do is get onto irc and talk directly to edson and I and we can walk you through classes.
http://www.jboss.org/drools/irc.html

Mark

Thanks in advance!

Cheers,
Leo.






On Sat, Sep 18, 2010 at 3:49 AM, Leonardo Gomes <[hidden email]> wrote:
if you look at the AbstractWorkingMemory insert method you'll see one argument is whether it's a logical insertion or not. You'll also see it check the global maintainTMS configuration and also retrieve the ObjectTypeConf. So between those things someone should be able to get it working.

Today, it enters a block where it operates on the equality map and also creates a default handle based on that TMS global option and *regardless* of whether it's a logical insert.

If I'm *not* putting things in the equality map for regular inserts, when a logical insert comes in, but there were already stated inserts, how will I know that? I would create a new handle for the logical insert and do the tms.addLogicalDependency(...), even tough there were regular inserts before and this seems to be a wrong behaviour.

Apparently, today, you can disable TMS and still use logical inserts in your drl, what, I believe, will lead to inconsistent behaviour, but you're at your own risk.

Ideas? I feel that I missed something :)

Cheers,
Leo.

P.S.: I reached the conclusions above based on the fact that LogicalAssertionTest started failing after I did the changes you suggested.

 


On Wed, Sep 15, 2010 at 3:47 PM, Mark Proctor <[hidden email]> wrote:
On 15/09/2010 14:35, Michael Anstis wrote:
Is this in drools-core; or drools-compiler?

Whilst not undertaking to do the work; have a purpose to nose through the code makes understanding easier.
It's all in DroolsCore.

It's a 5 minute hack for me and then 15 minute unit writing test. But I thought I'd write it up in a hope to bring someone else into the fold, we need more help writting the core engine someone else out there must want to work on current edge engine design :)

if you look at the AbstractWorkingMemory insert method you'll see one argument is whether it's a logical insertion or not. You'll also see it check the global maintainTMS configuration and also retrieve the ObjectTypeConf. So between those things someone should be able to get it working.

Mark


On 14 September 2010 16:47, Mark Proctor <[hidden email]> wrote:
Here is another project proposal, this time simpler. I think this one has Wolfgang's name on it ;)

http://blog.athico.com/2010/09/lazily-enabled-truth-maintenace.html

Three weeks ago I posted the project idea for "Left and Right Unlinking". So far there are no takers, so if you are interested let me know :)

In the meantime I tried to think of a simpler enhancement that we would like to see done.

At the moment Drools has a user setting "MaintainTMSOption" which can be true or false. It's a small optimisation that when turned off avoids using the equality hashmap that is maintained for all inserted objects.

It would be a much better idea to remove this configuration setting, thus simplifying things for end users and have TMS lazily enabled on demand.

For each object type there is an "ObjectTypeConf" configuration object that is retrieved every time a working memory action, such as insert, is executed. The enabledTMS boolean should be moved there, so there is one per object type, by default it is false.

When a working memory action occurs, like insert, it retrieved the ObjectTypeConf and checks the maintainTms boolean there, instead of the current engine scoped configuration. When a logical insertion occurs and the ObjectTypeConf is retrieved if maintainTms is false it sets the value to true and then iterates the associated ObjectTypeNode memory lazily adding all the objects to the TMS equality map. From then on for that ObjectType all inserted objects are added to that equality map.

With this you now have the advantage of TMS being laziy enabled, so the minor hashmap operation is no longer used and likewise a small memory saving from not populating the map. There is a further advantage that this is now fine grained and when enabled only impacts for that specific object type.

A further enhancement could use a int counter, instead of a boolean. Each logical insertion for that object type increases the counter, each retraction decreases the counter; even if automatically retracted if the truth is broken for that logical assertion. When the counter reaches zero, TMS for that OTN can be disabled. We do not however remove the objects from the equality map, as this would cause "churn" if TMS is continuously enabled and disabled. Instead when TMS is disabled record the current fact counter id. Then if TMS is disabled on a retraction but there is a counter id, we can check that counter id to see if the fact is prior to TMS being disabled and thus would need to be retracted from the equality map.

_______________________________________________
rules-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/rules-dev


_______________________________________________ rules-dev mailing list [hidden email] https://lists.jboss.org/mailman/listinfo/rules-dev


_______________________________________________
rules-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/rules-dev





_______________________________________________
rules-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/rules-dev
Reply | Threaded
Open this post in threaded view
|

Re: Lazily Enabled Truth Maintenace (Project Propsoal)

leo_gomes
Ok, it took me much longer that I had thought, but I think it's working now.. all tests passing except for ReteooWorkingMemoryTest that does:

        FactHandle fd = workingMemory.insert( string );

        assertEquals( 1,
                      tms.getAssertMap().size() );

Which is a good sign, since tms is now empty as long as you haven't done any logical insert :)

I implemented just the first part of your suggestion, with a boolean. I will try to implement the solution with the counter later this week and maybe do some adjustments before submitting a patch.

Do you have any suggestion in terms of unit tests that you would like to see or just the apparent lack of regressions is enough? Well, I can also think of some tests by taking ReteooWorkingMemoryTest as an example, in case you don't have anything particular in mind.

Cheers,
Leo.

 


On Mon, Sep 20, 2010 at 5:14 PM, Mark Proctor <[hidden email]> wrote:
On 20/09/2010 15:53, Leonardo Gomes wrote:
I think I wasn't quite clear in my last email, so let me try to reformulate it:

I also gave it a try to try to do what you suggested here (http://blog.athico.com/2010/09/lazily-enabled-truth-maintenace.html), Mark, and couldn't  make it work due to the following situation:

1) Current code seems to rely on the equality map to know that a logical insert (insertLogical) for an object that has already been regularly inserted (insert) should be ignored;
You would need to lazily maintain an equality map. When the first logical insertion is done we will have to first populate that map from the Object Type Node set of FactHandles.
In the "insert" one of the first lines is:
            ObjectTypeConf typeConf = this.typeConfReg.getObjectTypeConf( this.entryPoint,
                                                                          object );

So we get the ObjectTypeConf before we do anything with the object itself, and we can check if TMS is being maintained for that Object Type.


2) If I apply the modifications that you suggested, from what I understood, things would start to be put in the equality map only when a logical insert is issued;

*Problem*: How would I handle the situation described in item 1, if I don't have anything in the equality map at the moment a logical insert comes in and I have to "lazily activate" TMS?
final Rete source = this.ruleBase.getRete();
ClassObjectType cot = new ClassObjectType( MyClass.class );
Map<ObjectType, ObjectTypeNode> map = source.getObjectTypeNodes( EntryPoint.DEFAULT );
ObjectTypeNode node = map.get( cot );
final ObjectHashSet memory = (ObjectHashSet) workingMemory.getNodeMemory( node );

That "memory" is the set of currently asserted objects for that OTN. You can now iterate that and populate the equality hash map. This is a one off as the flag will be set on the ObjectTypeConf from then on and all objects will be added to the equality map at the point of insertion.

Does that help? All the information is there, you should have to know how to retrieve it :) We don't currently maintain TMS for anything other than the default entry point. Although I think that was a mistake and we will probably move all entrypoints to work with same, but for now you can ignore that and just focus on the default.

Btw this should not be confused with "equality" mode where the equality map has to be maintained by default from the start. Some would argue that a rule engine should only under work and understand on the bases of equality and thus all users must implement hashcode and equals correctly...

--

Moreover, I'm willing to attempt to implement the left and right un-linking and tried to start with this easier task to start to get familiar with drools-core. I already read the article you linked and your article. Would you have any document with an overview of the way drools implements rete?
not really no, Rete has already well documented in a number of papers. The best thing to do is get onto irc and talk directly to edson and I and we can walk you through classes.
http://www.jboss.org/drools/irc.html

Mark


Thanks in advance!

Cheers,
Leo.






On Sat, Sep 18, 2010 at 3:49 AM, Leonardo Gomes <[hidden email]> wrote:
if you look at the AbstractWorkingMemory insert method you'll see one argument is whether it's a logical insertion or not. You'll also see it check the global maintainTMS configuration and also retrieve the ObjectTypeConf. So between those things someone should be able to get it working.

Today, it enters a block where it operates on the equality map and also creates a default handle based on that TMS global option and *regardless* of whether it's a logical insert.

If I'm *not* putting things in the equality map for regular inserts, when a logical insert comes in, but there were already stated inserts, how will I know that? I would create a new handle for the logical insert and do the tms.addLogicalDependency(...), even tough there were regular inserts before and this seems to be a wrong behaviour.

Apparently, today, you can disable TMS and still use logical inserts in your drl, what, I believe, will lead to inconsistent behaviour, but you're at your own risk.

Ideas? I feel that I missed something :)

Cheers,
Leo.

P.S.: I reached the conclusions above based on the fact that LogicalAssertionTest started failing after I did the changes you suggested.

 


On Wed, Sep 15, 2010 at 3:47 PM, Mark Proctor <[hidden email]> wrote:
On 15/09/2010 14:35, Michael Anstis wrote:
Is this in drools-core; or drools-compiler?

Whilst not undertaking to do the work; have a purpose to nose through the code makes understanding easier.
It's all in DroolsCore.

It's a 5 minute hack for me and then 15 minute unit writing test. But I thought I'd write it up in a hope to bring someone else into the fold, we need more help writting the core engine someone else out there must want to work on current edge engine design :)

if you look at the AbstractWorkingMemory insert method you'll see one argument is whether it's a logical insertion or not. You'll also see it check the global maintainTMS configuration and also retrieve the ObjectTypeConf. So between those things someone should be able to get it working.

Mark


On 14 September 2010 16:47, Mark Proctor <[hidden email]> wrote:
Here is another project proposal, this time simpler. I think this one has Wolfgang's name on it ;)

http://blog.athico.com/2010/09/lazily-enabled-truth-maintenace.html

Three weeks ago I posted the project idea for "Left and Right Unlinking". So far there are no takers, so if you are interested let me know :)

In the meantime I tried to think of a simpler enhancement that we would like to see done.

At the moment Drools has a user setting "MaintainTMSOption" which can be true or false. It's a small optimisation that when turned off avoids using the equality hashmap that is maintained for all inserted objects.

It would be a much better idea to remove this configuration setting, thus simplifying things for end users and have TMS lazily enabled on demand.

For each object type there is an "ObjectTypeConf" configuration object that is retrieved every time a working memory action, such as insert, is executed. The enabledTMS boolean should be moved there, so there is one per object type, by default it is false.

When a working memory action occurs, like insert, it retrieved the ObjectTypeConf and checks the maintainTms boolean there, instead of the current engine scoped configuration. When a logical insertion occurs and the ObjectTypeConf is retrieved if maintainTms is false it sets the value to true and then iterates the associated ObjectTypeNode memory lazily adding all the objects to the TMS equality map. From then on for that ObjectType all inserted objects are added to that equality map.

With this you now have the advantage of TMS being laziy enabled, so the minor hashmap operation is no longer used and likewise a small memory saving from not populating the map. There is a further advantage that this is now fine grained and when enabled only impacts for that specific object type.

A further enhancement could use a int counter, instead of a boolean. Each logical insertion for that object type increases the counter, each retraction decreases the counter; even if automatically retracted if the truth is broken for that logical assertion. When the counter reaches zero, TMS for that OTN can be disabled. We do not however remove the objects from the equality map, as this would cause "churn" if TMS is continuously enabled and disabled. Instead when TMS is disabled record the current fact counter id. Then if TMS is disabled on a retraction but there is a counter id, we can check that counter id to see if the fact is prior to TMS being disabled and thus would need to be retracted from the equality map.

_______________________________________________
rules-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/rules-dev


_______________________________________________ rules-dev mailing list [hidden email] https://lists.jboss.org/mailman/listinfo/rules-dev


_______________________________________________
rules-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/rules-dev






_______________________________________________
rules-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/rules-dev
Reply | Threaded
Open this post in threaded view
|

Re: Lazily Enabled Truth Maintenace (Project Propsoal)

Mark Proctor
On 20/09/2010 21:58, Leonardo Gomes wrote:
Ok, it took me much longer that I had thought, but I think it's working now.. all tests passing except for ReteooWorkingMemoryTest that does:

        FactHandle fd = workingMemory.insert( string );

        assertEquals( 1,
                      tms.getAssertMap().size() );

Which is a good sign, since tms is now empty as long as you haven't done any logical insert :)

I implemented just the first part of your suggestion, with a boolean. I will try to implement the solution with the counter later this week and maybe do some adjustments before submitting a patch.

Do you have any suggestion in terms of unit tests that you would like to see or just the apparent lack of regressions is enough? Well, I can also think of some tests by taking ReteooWorkingMemoryTest as an example, in case you don't have anything particular in mind.
When you do some insertions cast to get some of the internal data structures and check the equality map is null, maybe check the OT confs are false, do a logical insertion check it can turn on for specific OTNs and not all and that it correctly adds all objects for the OTN. This will be important for the next stage of the work.

Mark

Cheers,
Leo.

 


On Mon, Sep 20, 2010 at 5:14 PM, Mark Proctor <[hidden email]> wrote:
On 20/09/2010 15:53, Leonardo Gomes wrote:
I think I wasn't quite clear in my last email, so let me try to reformulate it:

I also gave it a try to try to do what you suggested here (http://blog.athico.com/2010/09/lazily-enabled-truth-maintenace.html), Mark, and couldn't  make it work due to the following situation:

1) Current code seems to rely on the equality map to know that a logical insert (insertLogical) for an object that has already been regularly inserted (insert) should be ignored;
You would need to lazily maintain an equality map. When the first logical insertion is done we will have to first populate that map from the Object Type Node set of FactHandles.
In the "insert" one of the first lines is:
            ObjectTypeConf typeConf = this.typeConfReg.getObjectTypeConf( this.entryPoint,
                                                                          object );

So we get the ObjectTypeConf before we do anything with the object itself, and we can check if TMS is being maintained for that Object Type.


2) If I apply the modifications that you suggested, from what I understood, things would start to be put in the equality map only when a logical insert is issued;

*Problem*: How would I handle the situation described in item 1, if I don't have anything in the equality map at the moment a logical insert comes in and I have to "lazily activate" TMS?
final Rete source = this.ruleBase.getRete();
ClassObjectType cot = new ClassObjectType( MyClass.class );
Map<ObjectType, ObjectTypeNode> map = source.getObjectTypeNodes( EntryPoint.DEFAULT );
ObjectTypeNode node = map.get( cot );
final ObjectHashSet memory = (ObjectHashSet) workingMemory.getNodeMemory( node );

That "memory" is the set of currently asserted objects for that OTN. You can now iterate that and populate the equality hash map. This is a one off as the flag will be set on the ObjectTypeConf from then on and all objects will be added to the equality map at the point of insertion.

Does that help? All the information is there, you should have to know how to retrieve it :) We don't currently maintain TMS for anything other than the default entry point. Although I think that was a mistake and we will probably move all entrypoints to work with same, but for now you can ignore that and just focus on the default.

Btw this should not be confused with "equality" mode where the equality map has to be maintained by default from the start. Some would argue that a rule engine should only under work and understand on the bases of equality and thus all users must implement hashcode and equals correctly...

--

Moreover, I'm willing to attempt to implement the left and right un-linking and tried to start with this easier task to start to get familiar with drools-core. I already read the article you linked and your article. Would you have any document with an overview of the way drools implements rete?
not really no, Rete has already well documented in a number of papers. The best thing to do is get onto irc and talk directly to edson and I and we can walk you through classes.
http://www.jboss.org/drools/irc.html

Mark


Thanks in advance!

Cheers,
Leo.






On Sat, Sep 18, 2010 at 3:49 AM, Leonardo Gomes <[hidden email]> wrote:
if you look at the AbstractWorkingMemory insert method you'll see one argument is whether it's a logical insertion or not. You'll also see it check the global maintainTMS configuration and also retrieve the ObjectTypeConf. So between those things someone should be able to get it working.

Today, it enters a block where it operates on the equality map and also creates a default handle based on that TMS global option and *regardless* of whether it's a logical insert.

If I'm *not* putting things in the equality map for regular inserts, when a logical insert comes in, but there were already stated inserts, how will I know that? I would create a new handle for the logical insert and do the tms.addLogicalDependency(...), even tough there were regular inserts before and this seems to be a wrong behaviour.

Apparently, today, you can disable TMS and still use logical inserts in your drl, what, I believe, will lead to inconsistent behaviour, but you're at your own risk.

Ideas? I feel that I missed something :)

Cheers,
Leo.

P.S.: I reached the conclusions above based on the fact that LogicalAssertionTest started failing after I did the changes you suggested.

 


On Wed, Sep 15, 2010 at 3:47 PM, Mark Proctor <[hidden email]> wrote:
On 15/09/2010 14:35, Michael Anstis wrote:
Is this in drools-core; or drools-compiler?

Whilst not undertaking to do the work; have a purpose to nose through the code makes understanding easier.
It's all in DroolsCore.

It's a 5 minute hack for me and then 15 minute unit writing test. But I thought I'd write it up in a hope to bring someone else into the fold, we need more help writting the core engine someone else out there must want to work on current edge engine design :)

if you look at the AbstractWorkingMemory insert method you'll see one argument is whether it's a logical insertion or not. You'll also see it check the global maintainTMS configuration and also retrieve the ObjectTypeConf. So between those things someone should be able to get it working.

Mark


On 14 September 2010 16:47, Mark Proctor <[hidden email]> wrote:
Here is another project proposal, this time simpler. I think this one has Wolfgang's name on it ;)

http://blog.athico.com/2010/09/lazily-enabled-truth-maintenace.html

Three weeks ago I posted the project idea for "Left and Right Unlinking". So far there are no takers, so if you are interested let me know :)

In the meantime I tried to think of a simpler enhancement that we would like to see done.

At the moment Drools has a user setting "MaintainTMSOption" which can be true or false. It's a small optimisation that when turned off avoids using the equality hashmap that is maintained for all inserted objects.

It would be a much better idea to remove this configuration setting, thus simplifying things for end users and have TMS lazily enabled on demand.

For each object type there is an "ObjectTypeConf" configuration object that is retrieved every time a working memory action, such as insert, is executed. The enabledTMS boolean should be moved there, so there is one per object type, by default it is false.

When a working memory action occurs, like insert, it retrieved the ObjectTypeConf and checks the maintainTms boolean there, instead of the current engine scoped configuration. When a logical insertion occurs and the ObjectTypeConf is retrieved if maintainTms is false it sets the value to true and then iterates the associated ObjectTypeNode memory lazily adding all the objects to the TMS equality map. From then on for that ObjectType all inserted objects are added to that equality map.

With this you now have the advantage of TMS being laziy enabled, so the minor hashmap operation is no longer used and likewise a small memory saving from not populating the map. There is a further advantage that this is now fine grained and when enabled only impacts for that specific object type.

A further enhancement could use a int counter, instead of a boolean. Each logical insertion for that object type increases the counter, each retraction decreases the counter; even if automatically retracted if the truth is broken for that logical assertion. When the counter reaches zero, TMS for that OTN can be disabled. We do not however remove the objects from the equality map, as this would cause "churn" if TMS is continuously enabled and disabled. Instead when TMS is disabled record the current fact counter id. Then if TMS is disabled on a retraction but there is a counter id, we can check that counter id to see if the fact is prior to TMS being disabled and thus would need to be retracted from the equality map.

_______________________________________________
rules-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/rules-dev


_______________________________________________ rules-dev mailing list [hidden email] https://lists.jboss.org/mailman/listinfo/rules-dev


_______________________________________________
rules-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/rules-dev







_______________________________________________
rules-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/rules-dev
Reply | Threaded
Open this post in threaded view
|

Re: Lazily Enabled Truth Maintenace (Project Propsoal)

salaboy
nice, is it working? I would like to see the changes in the commits..
Greetings.

On Mon, Sep 20, 2010 at 7:09 PM, Mark Proctor <[hidden email]> wrote:
On 20/09/2010 21:58, Leonardo Gomes wrote:
Ok, it took me much longer that I had thought, but I think it's working now.. all tests passing except for ReteooWorkingMemoryTest that does:

        FactHandle fd = workingMemory.insert( string );

        assertEquals( 1,
                      tms.getAssertMap().size() );

Which is a good sign, since tms is now empty as long as you haven't done any logical insert :)

I implemented just the first part of your suggestion, with a boolean. I will try to implement the solution with the counter later this week and maybe do some adjustments before submitting a patch.

Do you have any suggestion in terms of unit tests that you would like to see or just the apparent lack of regressions is enough? Well, I can also think of some tests by taking ReteooWorkingMemoryTest as an example, in case you don't have anything particular in mind.
When you do some insertions cast to get some of the internal data structures and check the equality map is null, maybe check the OT confs are false, do a logical insertion check it can turn on for specific OTNs and not all and that it correctly adds all objects for the OTN. This will be important for the next stage of the work.

Mark


Cheers,
Leo.

 


On Mon, Sep 20, 2010 at 5:14 PM, Mark Proctor <[hidden email]> wrote:
On 20/09/2010 15:53, Leonardo Gomes wrote:
I think I wasn't quite clear in my last email, so let me try to reformulate it:

I also gave it a try to try to do what you suggested here (http://blog.athico.com/2010/09/lazily-enabled-truth-maintenace.html), Mark, and couldn't  make it work due to the following situation:

1) Current code seems to rely on the equality map to know that a logical insert (insertLogical) for an object that has already been regularly inserted (insert) should be ignored;
You would need to lazily maintain an equality map. When the first logical insertion is done we will have to first populate that map from the Object Type Node set of FactHandles.
In the "insert" one of the first lines is:
            ObjectTypeConf typeConf = this.typeConfReg.getObjectTypeConf( this.entryPoint,
                                                                          object );

So we get the ObjectTypeConf before we do anything with the object itself, and we can check if TMS is being maintained for that Object Type.


2) If I apply the modifications that you suggested, from what I understood, things would start to be put in the equality map only when a logical insert is issued;

*Problem*: How would I handle the situation described in item 1, if I don't have anything in the equality map at the moment a logical insert comes in and I have to "lazily activate" TMS?
final Rete source = this.ruleBase.getRete();
ClassObjectType cot = new ClassObjectType( MyClass.class );
Map<ObjectType, ObjectTypeNode> map = source.getObjectTypeNodes( EntryPoint.DEFAULT );
ObjectTypeNode node = map.get( cot );
final ObjectHashSet memory = (ObjectHashSet) workingMemory.getNodeMemory( node );

That "memory" is the set of currently asserted objects for that OTN. You can now iterate that and populate the equality hash map. This is a one off as the flag will be set on the ObjectTypeConf from then on and all objects will be added to the equality map at the point of insertion.

Does that help? All the information is there, you should have to know how to retrieve it :) We don't currently maintain TMS for anything other than the default entry point. Although I think that was a mistake and we will probably move all entrypoints to work with same, but for now you can ignore that and just focus on the default.

Btw this should not be confused with "equality" mode where the equality map has to be maintained by default from the start. Some would argue that a rule engine should only under work and understand on the bases of equality and thus all users must implement hashcode and equals correctly...

--

Moreover, I'm willing to attempt to implement the left and right un-linking and tried to start with this easier task to start to get familiar with drools-core. I already read the article you linked and your article. Would you have any document with an overview of the way drools implements rete?
not really no, Rete has already well documented in a number of papers. The best thing to do is get onto irc and talk directly to edson and I and we can walk you through classes.
http://www.jboss.org/drools/irc.html

Mark


Thanks in advance!

Cheers,
Leo.






On Sat, Sep 18, 2010 at 3:49 AM, Leonardo Gomes <[hidden email]> wrote:
if you look at the AbstractWorkingMemory insert method you'll see one argument is whether it's a logical insertion or not. You'll also see it check the global maintainTMS configuration and also retrieve the ObjectTypeConf. So between those things someone should be able to get it working.

Today, it enters a block where it operates on the equality map and also creates a default handle based on that TMS global option and *regardless* of whether it's a logical insert.

If I'm *not* putting things in the equality map for regular inserts, when a logical insert comes in, but there were already stated inserts, how will I know that? I would create a new handle for the logical insert and do the tms.addLogicalDependency(...), even tough there were regular inserts before and this seems to be a wrong behaviour.

Apparently, today, you can disable TMS and still use logical inserts in your drl, what, I believe, will lead to inconsistent behaviour, but you're at your own risk.

Ideas? I feel that I missed something :)

Cheers,
Leo.

P.S.: I reached the conclusions above based on the fact that LogicalAssertionTest started failing after I did the changes you suggested.

 


On Wed, Sep 15, 2010 at 3:47 PM, Mark Proctor <[hidden email]> wrote:
On 15/09/2010 14:35, Michael Anstis wrote:
Is this in drools-core; or drools-compiler?

Whilst not undertaking to do the work; have a purpose to nose through the code makes understanding easier.
It's all in DroolsCore.

It's a 5 minute hack for me and then 15 minute unit writing test. But I thought I'd write it up in a hope to bring someone else into the fold, we need more help writting the core engine someone else out there must want to work on current edge engine design :)

if you look at the AbstractWorkingMemory insert method you'll see one argument is whether it's a logical insertion or not. You'll also see it check the global maintainTMS configuration and also retrieve the ObjectTypeConf. So between those things someone should be able to get it working.

Mark


On 14 September 2010 16:47, Mark Proctor <[hidden email]> wrote:
Here is another project proposal, this time simpler. I think this one has Wolfgang's name on it ;)

http://blog.athico.com/2010/09/lazily-enabled-truth-maintenace.html

Three weeks ago I posted the project idea for "Left and Right Unlinking". So far there are no takers, so if you are interested let me know :)

In the meantime I tried to think of a simpler enhancement that we would like to see done.

At the moment Drools has a user setting "MaintainTMSOption" which can be true or false. It's a small optimisation that when turned off avoids using the equality hashmap that is maintained for all inserted objects.

It would be a much better idea to remove this configuration setting, thus simplifying things for end users and have TMS lazily enabled on demand.

For each object type there is an "ObjectTypeConf" configuration object that is retrieved every time a working memory action, such as insert, is executed. The enabledTMS boolean should be moved there, so there is one per object type, by default it is false.

When a working memory action occurs, like insert, it retrieved the ObjectTypeConf and checks the maintainTms boolean there, instead of the current engine scoped configuration. When a logical insertion occurs and the ObjectTypeConf is retrieved if maintainTms is false it sets the value to true and then iterates the associated ObjectTypeNode memory lazily adding all the objects to the TMS equality map. From then on for that ObjectType all inserted objects are added to that equality map.

With this you now have the advantage of TMS being laziy enabled, so the minor hashmap operation is no longer used and likewise a small memory saving from not populating the map. There is a further advantage that this is now fine grained and when enabled only impacts for that specific object type.

A further enhancement could use a int counter, instead of a boolean. Each logical insertion for that object type increases the counter, each retraction decreases the counter; even if automatically retracted if the truth is broken for that logical assertion. When the counter reaches zero, TMS for that OTN can be disabled. We do not however remove the objects from the equality map, as this would cause "churn" if TMS is continuously enabled and disabled. Instead when TMS is disabled record the current fact counter id. Then if TMS is disabled on a retraction but there is a counter id, we can check that counter id to see if the fact is prior to TMS being disabled and thus would need to be retracted from the equality map.

_______________________________________________
rules-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/rules-dev


_______________________________________________ rules-dev mailing list [hidden email] https://lists.jboss.org/mailman/listinfo/rules-dev


_______________________________________________
rules-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/rules-dev







_______________________________________________
rules-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/rules-dev




--
 - CTO @ http://www.plugtree.com 
 - MyJourney @ http://salaboy.wordpress.com
 - Co-Founder @ http://www.jbug.com.ar
 
 - Salatino "Salaboy" Mauricio -

_______________________________________________
rules-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/rules-dev
Reply | Threaded
Open this post in threaded view
|

Re: Lazily Enabled Truth Maintenace (Project Propsoal)

Anstis, Michael (M.)
You're not alone in wanting to see the changes...
 
It'd be good if Leo would be willing to post his experience as a comment on the original blog http://blog.athico.com/2010/09/lazily-enabled-truth-maintenace.html
 
It might help others wanting to get involved.


From: [hidden email] [mailto:[hidden email]] On Behalf Of Mauricio Salatino
Sent: 20 September 2010 23:18
To: Rules Dev List
Subject: Re: [rules-dev] Lazily Enabled Truth Maintenace (Project Propsoal)

nice, is it working? I would like to see the changes in the commits..
Greetings.

On Mon, Sep 20, 2010 at 7:09 PM, Mark Proctor <[hidden email]> wrote:
On 20/09/2010 21:58, Leonardo Gomes wrote:
Ok, it took me much longer that I had thought, but I think it's working now.. all tests passing except for ReteooWorkingMemoryTest that does:

        FactHandle fd = workingMemory.insert( string );

        assertEquals( 1,
                      tms.getAssertMap().size() );

Which is a good sign, since tms is now empty as long as you haven't done any logical insert :)

I implemented just the first part of your suggestion, with a boolean. I will try to implement the solution with the counter later this week and maybe do some adjustments before submitting a patch.

Do you have any suggestion in terms of unit tests that you would like to see or just the apparent lack of regressions is enough? Well, I can also think of some tests by taking ReteooWorkingMemoryTest as an example, in case you don't have anything particular in mind.
When you do some insertions cast to get some of the internal data structures and check the equality map is null, maybe check the OT confs are false, do a logical insertion check it can turn on for specific OTNs and not all and that it correctly adds all objects for the OTN. This will be important for the next stage of the work.

Mark


Cheers,
Leo.

 


On Mon, Sep 20, 2010 at 5:14 PM, Mark Proctor <[hidden email]> wrote:
On 20/09/2010 15:53, Leonardo Gomes wrote:
I think I wasn't quite clear in my last email, so let me try to reformulate it:

I also gave it a try to try to do what you suggested here (http://blog.athico.com/2010/09/lazily-enabled-truth-maintenace.html), Mark, and couldn't  make it work due to the following situation:

1) Current code seems to rely on the equality map to know that a logical insert (insertLogical) for an object that has already been regularly inserted (insert) should be ignored;
You would need to lazily maintain an equality map. When the first logical insertion is done we will have to first populate that map from the Object Type Node set of FactHandles.
In the "insert" one of the first lines is:
            ObjectTypeConf typeConf = this.typeConfReg.getObjectTypeConf( this.entryPoint,
                                                                          object );

So we get the ObjectTypeConf before we do anything with the object itself, and we can check if TMS is being maintained for that Object Type.


2) If I apply the modifications that you suggested, from what I understood, things would start to be put in the equality map only when a logical insert is issued;

*Problem*: How would I handle the situation described in item 1, if I don't have anything in the equality map at the moment a logical insert comes in and I have to "lazily activate" TMS?
final Rete source = this.ruleBase.getRete();
ClassObjectType cot = new ClassObjectType( MyClass.class );
Map<ObjectType, ObjectTypeNode> map = source.getObjectTypeNodes( EntryPoint.DEFAULT );
ObjectTypeNode node = map.get( cot );
final ObjectHashSet memory = (ObjectHashSet) workingMemory.getNodeMemory( node );

That "memory" is the set of currently asserted objects for that OTN. You can now iterate that and populate the equality hash map. This is a one off as the flag will be set on the ObjectTypeConf from then on and all objects will be added to the equality map at the point of insertion.

Does that help? All the information is there, you should have to know how to retrieve it :) We don't currently maintain TMS for anything other than the default entry point. Although I think that was a mistake and we will probably move all entrypoints to work with same, but for now you can ignore that and just focus on the default.

Btw this should not be confused with "equality" mode where the equality map has to be maintained by default from the start. Some would argue that a rule engine should only under work and understand on the bases of equality and thus all users must implement hashcode and equals correctly...

--

Moreover, I'm willing to attempt to implement the left and right un-linking and tried to start with this easier task to start to get familiar with drools-core. I already read the article you linked and your article. Would you have any document with an overview of the way drools implements rete?
not really no, Rete has already well documented in a number of papers. The best thing to do is get onto irc and talk directly to edson and I and we can walk you through classes.
http://www.jboss.org/drools/irc.html

Mark


Thanks in advance!

Cheers,
Leo.






On Sat, Sep 18, 2010 at 3:49 AM, Leonardo Gomes <[hidden email]> wrote:
if you look at the AbstractWorkingMemory insert method you'll see one argument is whether it's a logical insertion or not. You'll also see it check the global maintainTMS configuration and also retrieve the ObjectTypeConf. So between those things someone should be able to get it working.

Today, it enters a block where it operates on the equality map and also creates a default handle based on that TMS global option and *regardless* of whether it's a logical insert.

If I'm *not* putting things in the equality map for regular inserts, when a logical insert comes in, but there were already stated inserts, how will I know that? I would create a new handle for the logical insert and do the tms.addLogicalDependency(...), even tough there were regular inserts before and this seems to be a wrong behaviour.

Apparently, today, you can disable TMS and still use logical inserts in your drl, what, I believe, will lead to inconsistent behaviour, but you're at your own risk.

Ideas? I feel that I missed something :)

Cheers,
Leo.

P.S.: I reached the conclusions above based on the fact that LogicalAssertionTest started failing after I did the changes you suggested.

 


On Wed, Sep 15, 2010 at 3:47 PM, Mark Proctor <[hidden email]> wrote:
On 15/09/2010 14:35, Michael Anstis wrote:
Is this in drools-core; or drools-compiler?

Whilst not undertaking to do the work; have a purpose to nose through the code makes understanding easier.
It's all in DroolsCore.

It's a 5 minute hack for me and then 15 minute unit writing test. But I thought I'd write it up in a hope to bring someone else into the fold, we need more help writting the core engine someone else out there must want to work on current edge engine design :)

if you look at the AbstractWorkingMemory insert method you'll see one argument is whether it's a logical insertion or not. You'll also see it check the global maintainTMS configuration and also retrieve the ObjectTypeConf. So between those things someone should be able to get it working.

Mark


On 14 September 2010 16:47, Mark Proctor <[hidden email]> wrote:
Here is another project proposal, this time simpler. I think this one has Wolfgang's name on it ;)

http://blog.athico.com/2010/09/lazily-enabled-truth-maintenace.html

Three weeks ago I posted the project idea for "Left and Right Unlinking". So far there are no takers, so if you are interested let me know :)

In the meantime I tried to think of a simpler enhancement that we would like to see done.

At the moment Drools has a user setting "MaintainTMSOption" which can be true or false. It's a small optimisation that when turned off avoids using the equality hashmap that is maintained for all inserted objects.

It would be a much better idea to remove this configuration setting, thus simplifying things for end users and have TMS lazily enabled on demand.

For each object type there is an "ObjectTypeConf" configuration object that is retrieved every time a working memory action, such as insert, is executed. The enabledTMS boolean should be moved there, so there is one per object type, by default it is false.

When a working memory action occurs, like insert, it retrieved the ObjectTypeConf and checks the maintainTms boolean there, instead of the current engine scoped configuration. When a logical insertion occurs and the ObjectTypeConf is retrieved if maintainTms is false it sets the value to true and then iterates the associated ObjectTypeNode memory lazily adding all the objects to the TMS equality map. From then on for that ObjectType all inserted objects are added to that equality map.

With this you now have the advantage of TMS being laziy enabled, so the minor hashmap operation is no longer used and likewise a small memory saving from not populating the map. There is a further advantage that this is now fine grained and when enabled only impacts for that specific object type.

A further enhancement could use a int counter, instead of a boolean. Each logical insertion for that object type increases the counter, each retraction decreases the counter; even if automatically retracted if the truth is broken for that logical assertion. When the counter reaches zero, TMS for that OTN can be disabled. We do not however remove the objects from the equality map, as this would cause "churn" if TMS is continuously enabled and disabled. Instead when TMS is disabled record the current fact counter id. Then if TMS is disabled on a retraction but there is a counter id, we can check that counter id to see if the fact is prior to TMS being disabled and thus would need to be retracted from the equality map.

_______________________________________________
rules-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/rules-dev


_______________________________________________ rules-dev mailing list [hidden email] https://lists.jboss.org/mailman/listinfo/rules-dev


_______________________________________________
rules-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/rules-dev







_______________________________________________
rules-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/rules-dev




--
 - CTO @ http://www.plugtree.com 
 - MyJourney @ http://salaboy.wordpress.com
 - Co-Founder @ http://www.jbug.com.ar
 
 - Salatino "Salaboy" Mauricio -

_______________________________________________
rules-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/rules-dev
Reply | Threaded
Open this post in threaded view
|

Re: Lazily Enabled Truth Maintenace (Project Propsoal)

leo_gomes
In reply to this post by Mark Proctor
Hi,

I implemented the test cases you suggested, Mark, and started working on the counter. I discussed a bit with Edson yesterday and he clarified the difference between the counter for logically inserted facts and the fact id counter. This part seems to be fine, but I have a problem with the following scenario:

After an initial activation of TMS I put everything in the equality map, then, due to retractions, the counter gets back to zero and I have to disable TMS for that specific type. When a new fact is logically inserted I have to look at the OTN memory and get everything that was inserted there since the last deactivation of TMS, to be be able to re-enable it. To do so I use the fact handle id that I'm supposed to have stored together with the counter. Fine.

Problem: Since the OTN memory is an ObjectHashSet, I will have to iterate through it and filter (there's an ObjectFilter interface that I think I can implement) all the elements that have id greater than the one I stored. This will probably be inefficient and still likely to cause churn.

Possible solutions:

1) Use a TreeMap? ** Don't think so, HashSet retrieval and insertion is O(1), on average, whereas TreeMap is O(log(n));

2) Use a LinkedHashSet (probably a homegrown implementation on top of ObjectHashSet)? ** Would allow me to get the ObjectEntry I want and then call getNext() up to the end, inserting all of them in the equality map;

3) Never disable TMS for a given type once it has been enabled? ** I think that lazily enabling TMS per type might be already enough an optimization, in a sense that the overhead with the calculation of the delta and all this logic to disable/re-enable might not be worthy. Moreover, switching from HashSet to LinkedHashSet may have some disadvantages.

What do you think?

Cheers,
Leo.


On Tue, Sep 21, 2010 at 12:09 AM, Mark Proctor <[hidden email]> wrote:
On 20/09/2010 21:58, Leonardo Gomes wrote:
Ok, it took me much longer that I had thought, but I think it's working now.. all tests passing except for ReteooWorkingMemoryTest that does:

        FactHandle fd = workingMemory.insert( string );

        assertEquals( 1,
                      tms.getAssertMap().size() );

Which is a good sign, since tms is now empty as long as you haven't done any logical insert :)

I implemented just the first part of your suggestion, with a boolean. I will try to implement the solution with the counter later this week and maybe do some adjustments before submitting a patch.

Do you have any suggestion in terms of unit tests that you would like to see or just the apparent lack of regressions is enough? Well, I can also think of some tests by taking ReteooWorkingMemoryTest as an example, in case you don't have anything particular in mind.
When you do some insertions cast to get some of the internal data structures and check the equality map is null, maybe check the OT confs are false, do a logical insertion check it can turn on for specific OTNs and not all and that it correctly adds all objects for the OTN. This will be important for the next stage of the work.

Mark


Cheers,
Leo.

 


On Mon, Sep 20, 2010 at 5:14 PM, Mark Proctor <[hidden email]> wrote:
On 20/09/2010 15:53, Leonardo Gomes wrote:
I think I wasn't quite clear in my last email, so let me try to reformulate it:

I also gave it a try to try to do what you suggested here (http://blog.athico.com/2010/09/lazily-enabled-truth-maintenace.html), Mark, and couldn't  make it work due to the following situation:

1) Current code seems to rely on the equality map to know that a logical insert (insertLogical) for an object that has already been regularly inserted (insert) should be ignored;
You would need to lazily maintain an equality map. When the first logical insertion is done we will have to first populate that map from the Object Type Node set of FactHandles.
In the "insert" one of the first lines is:
            ObjectTypeConf typeConf = this.typeConfReg.getObjectTypeConf( this.entryPoint,
                                                                          object );

So we get the ObjectTypeConf before we do anything with the object itself, and we can check if TMS is being maintained for that Object Type.


2) If I apply the modifications that you suggested, from what I understood, things would start to be put in the equality map only when a logical insert is issued;

*Problem*: How would I handle the situation described in item 1, if I don't have anything in the equality map at the moment a logical insert comes in and I have to "lazily activate" TMS?
final Rete source = this.ruleBase.getRete();
ClassObjectType cot = new ClassObjectType( MyClass.class );
Map<ObjectType, ObjectTypeNode> map = source.getObjectTypeNodes( EntryPoint.DEFAULT );
ObjectTypeNode node = map.get( cot );
final ObjectHashSet memory = (ObjectHashSet) workingMemory.getNodeMemory( node );

That "memory" is the set of currently asserted objects for that OTN. You can now iterate that and populate the equality hash map. This is a one off as the flag will be set on the ObjectTypeConf from then on and all objects will be added to the equality map at the point of insertion.

Does that help? All the information is there, you should have to know how to retrieve it :) We don't currently maintain TMS for anything other than the default entry point. Although I think that was a mistake and we will probably move all entrypoints to work with same, but for now you can ignore that and just focus on the default.

Btw this should not be confused with "equality" mode where the equality map has to be maintained by default from the start. Some would argue that a rule engine should only under work and understand on the bases of equality and thus all users must implement hashcode and equals correctly...

--

Moreover, I'm willing to attempt to implement the left and right un-linking and tried to start with this easier task to start to get familiar with drools-core. I already read the article you linked and your article. Would you have any document with an overview of the way drools implements rete?
not really no, Rete has already well documented in a number of papers. The best thing to do is get onto irc and talk directly to edson and I and we can walk you through classes.
http://www.jboss.org/drools/irc.html

Mark


Thanks in advance!

Cheers,
Leo.






On Sat, Sep 18, 2010 at 3:49 AM, Leonardo Gomes <[hidden email]> wrote:
if you look at the AbstractWorkingMemory insert method you'll see one argument is whether it's a logical insertion or not. You'll also see it check the global maintainTMS configuration and also retrieve the ObjectTypeConf. So between those things someone should be able to get it working.

Today, it enters a block where it operates on the equality map and also creates a default handle based on that TMS global option and *regardless* of whether it's a logical insert.

If I'm *not* putting things in the equality map for regular inserts, when a logical insert comes in, but there were already stated inserts, how will I know that? I would create a new handle for the logical insert and do the tms.addLogicalDependency(...), even tough there were regular inserts before and this seems to be a wrong behaviour.

Apparently, today, you can disable TMS and still use logical inserts in your drl, what, I believe, will lead to inconsistent behaviour, but you're at your own risk.

Ideas? I feel that I missed something :)

Cheers,
Leo.

P.S.: I reached the conclusions above based on the fact that LogicalAssertionTest started failing after I did the changes you suggested.

 


On Wed, Sep 15, 2010 at 3:47 PM, Mark Proctor <[hidden email]> wrote:
On 15/09/2010 14:35, Michael Anstis wrote:
Is this in drools-core; or drools-compiler?

Whilst not undertaking to do the work; have a purpose to nose through the code makes understanding easier.
It's all in DroolsCore.

It's a 5 minute hack for me and then 15 minute unit writing test. But I thought I'd write it up in a hope to bring someone else into the fold, we need more help writting the core engine someone else out there must want to work on current edge engine design :)

if you look at the AbstractWorkingMemory insert method you'll see one argument is whether it's a logical insertion or not. You'll also see it check the global maintainTMS configuration and also retrieve the ObjectTypeConf. So between those things someone should be able to get it working.

Mark


On 14 September 2010 16:47, Mark Proctor <[hidden email]> wrote:
Here is another project proposal, this time simpler. I think this one has Wolfgang's name on it ;)

http://blog.athico.com/2010/09/lazily-enabled-truth-maintenace.html

Three weeks ago I posted the project idea for "Left and Right Unlinking". So far there are no takers, so if you are interested let me know :)

In the meantime I tried to think of a simpler enhancement that we would like to see done.

At the moment Drools has a user setting "MaintainTMSOption" which can be true or false. It's a small optimisation that when turned off avoids using the equality hashmap that is maintained for all inserted objects.

It would be a much better idea to remove this configuration setting, thus simplifying things for end users and have TMS lazily enabled on demand.

For each object type there is an "ObjectTypeConf" configuration object that is retrieved every time a working memory action, such as insert, is executed. The enabledTMS boolean should be moved there, so there is one per object type, by default it is false.

When a working memory action occurs, like insert, it retrieved the ObjectTypeConf and checks the maintainTms boolean there, instead of the current engine scoped configuration. When a logical insertion occurs and the ObjectTypeConf is retrieved if maintainTms is false it sets the value to true and then iterates the associated ObjectTypeNode memory lazily adding all the objects to the TMS equality map. From then on for that ObjectType all inserted objects are added to that equality map.

With this you now have the advantage of TMS being laziy enabled, so the minor hashmap operation is no longer used and likewise a small memory saving from not populating the map. There is a further advantage that this is now fine grained and when enabled only impacts for that specific object type.

A further enhancement could use a int counter, instead of a boolean. Each logical insertion for that object type increases the counter, each retraction decreases the counter; even if automatically retracted if the truth is broken for that logical assertion. When the counter reaches zero, TMS for that OTN can be disabled. We do not however remove the objects from the equality map, as this would cause "churn" if TMS is continuously enabled and disabled. Instead when TMS is disabled record the current fact counter id. Then if TMS is disabled on a retraction but there is a counter id, we can check that counter id to see if the fact is prior to TMS being disabled and thus would need to be retracted from the equality map.

_______________________________________________
rules-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/rules-dev


_______________________________________________ rules-dev mailing list [hidden email] https://lists.jboss.org/mailman/listinfo/rules-dev


_______________________________________________
rules-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/rules-dev








_______________________________________________
rules-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/rules-dev
Reply | Threaded
Open this post in threaded view
|

Re: Lazily Enabled Truth Maintenace (Project Propsoal)

leo_gomes
In reply to this post by Anstis, Michael (M.)
Hi Michael,

Yes, I can do so, once this is finished :)

Cheers,
Leo.


On Tue, Sep 21, 2010 at 10:13 AM, Anstis, Michael (M.) <[hidden email]> wrote:
You're not alone in wanting to see the changes...
 
It'd be good if Leo would be willing to post his experience as a comment on the original blog http://blog.athico.com/2010/09/lazily-enabled-truth-maintenace.html
 
It might help others wanting to get involved.


From: [hidden email] [mailto:[hidden email]] On Behalf Of Mauricio Salatino
Sent: 20 September 2010 23:18
To: Rules Dev List
Subject: Re: [rules-dev] Lazily Enabled Truth Maintenace (Project Propsoal)

nice, is it working? I would like to see the changes in the commits..
Greetings.

On Mon, Sep 20, 2010 at 7:09 PM, Mark Proctor <[hidden email]> wrote:
On 20/09/2010 21:58, Leonardo Gomes wrote:
Ok, it took me much longer that I had thought, but I think it's working now.. all tests passing except for ReteooWorkingMemoryTest that does:

        FactHandle fd = workingMemory.insert( string );

        assertEquals( 1,
                      tms.getAssertMap().size() );

Which is a good sign, since tms is now empty as long as you haven't done any logical insert :)

I implemented just the first part of your suggestion, with a boolean. I will try to implement the solution with the counter later this week and maybe do some adjustments before submitting a patch.

Do you have any suggestion in terms of unit tests that you would like to see or just the apparent lack of regressions is enough? Well, I can also think of some tests by taking ReteooWorkingMemoryTest as an example, in case you don't have anything particular in mind.
When you do some insertions cast to get some of the internal data structures and check the equality map is null, maybe check the OT confs are false, do a logical insertion check it can turn on for specific OTNs and not all and that it correctly adds all objects for the OTN. This will be important for the next stage of the work.

Mark


Cheers,
Leo.

 


On Mon, Sep 20, 2010 at 5:14 PM, Mark Proctor <[hidden email]> wrote:
On 20/09/2010 15:53, Leonardo Gomes wrote:
I think I wasn't quite clear in my last email, so let me try to reformulate it:

I also gave it a try to try to do what you suggested here (http://blog.athico.com/2010/09/lazily-enabled-truth-maintenace.html), Mark, and couldn't  make it work due to the following situation:

1) Current code seems to rely on the equality map to know that a logical insert (insertLogical) for an object that has already been regularly inserted (insert) should be ignored;
You would need to lazily maintain an equality map. When the first logical insertion is done we will have to first populate that map from the Object Type Node set of FactHandles.
In the "insert" one of the first lines is:
            ObjectTypeConf typeConf = this.typeConfReg.getObjectTypeConf( this.entryPoint,
                                                                          object );

So we get the ObjectTypeConf before we do anything with the object itself, and we can check if TMS is being maintained for that Object Type.


2) If I apply the modifications that you suggested, from what I understood, things would start to be put in the equality map only when a logical insert is issued;

*Problem*: How would I handle the situation described in item 1, if I don't have anything in the equality map at the moment a logical insert comes in and I have to "lazily activate" TMS?
final Rete source = this.ruleBase.getRete();
ClassObjectType cot = new ClassObjectType( MyClass.class );
Map<ObjectType, ObjectTypeNode> map = source.getObjectTypeNodes( EntryPoint.DEFAULT );
ObjectTypeNode node = map.get( cot );
final ObjectHashSet memory = (ObjectHashSet) workingMemory.getNodeMemory( node );

That "memory" is the set of currently asserted objects for that OTN. You can now iterate that and populate the equality hash map. This is a one off as the flag will be set on the ObjectTypeConf from then on and all objects will be added to the equality map at the point of insertion.

Does that help? All the information is there, you should have to know how to retrieve it :) We don't currently maintain TMS for anything other than the default entry point. Although I think that was a mistake and we will probably move all entrypoints to work with same, but for now you can ignore that and just focus on the default.

Btw this should not be confused with "equality" mode where the equality map has to be maintained by default from the start. Some would argue that a rule engine should only under work and understand on the bases of equality and thus all users must implement hashcode and equals correctly...

--

Moreover, I'm willing to attempt to implement the left and right un-linking and tried to start with this easier task to start to get familiar with drools-core. I already read the article you linked and your article. Would you have any document with an overview of the way drools implements rete?
not really no, Rete has already well documented in a number of papers. The best thing to do is get onto irc and talk directly to edson and I and we can walk you through classes.
http://www.jboss.org/drools/irc.html

Mark


Thanks in advance!

Cheers,
Leo.






On Sat, Sep 18, 2010 at 3:49 AM, Leonardo Gomes <[hidden email]> wrote:
if you look at the AbstractWorkingMemory insert method you'll see one argument is whether it's a logical insertion or not. You'll also see it check the global maintainTMS configuration and also retrieve the ObjectTypeConf. So between those things someone should be able to get it working.

Today, it enters a block where it operates on the equality map and also creates a default handle based on that TMS global option and *regardless* of whether it's a logical insert.

If I'm *not* putting things in the equality map for regular inserts, when a logical insert comes in, but there were already stated inserts, how will I know that? I would create a new handle for the logical insert and do the tms.addLogicalDependency(...), even tough there were regular inserts before and this seems to be a wrong behaviour.

Apparently, today, you can disable TMS and still use logical inserts in your drl, what, I believe, will lead to inconsistent behaviour, but you're at your own risk.

Ideas? I feel that I missed something :)

Cheers,
Leo.

P.S.: I reached the conclusions above based on the fact that LogicalAssertionTest started failing after I did the changes you suggested.

 


On Wed, Sep 15, 2010 at 3:47 PM, Mark Proctor <[hidden email]> wrote:
On 15/09/2010 14:35, Michael Anstis wrote:
Is this in drools-core; or drools-compiler?

Whilst not undertaking to do the work; have a purpose to nose through the code makes understanding easier.
It's all in DroolsCore.

It's a 5 minute hack for me and then 15 minute unit writing test. But I thought I'd write it up in a hope to bring someone else into the fold, we need more help writting the core engine someone else out there must want to work on current edge engine design :)

if you look at the AbstractWorkingMemory insert method you'll see one argument is whether it's a logical insertion or not. You'll also see it check the global maintainTMS configuration and also retrieve the ObjectTypeConf. So between those things someone should be able to get it working.

Mark


On 14 September 2010 16:47, Mark Proctor <[hidden email]> wrote:
Here is another project proposal, this time simpler. I think this one has Wolfgang's name on it ;)

http://blog.athico.com/2010/09/lazily-enabled-truth-maintenace.html

Three weeks ago I posted the project idea for "Left and Right Unlinking". So far there are no takers, so if you are interested let me know :)

In the meantime I tried to think of a simpler enhancement that we would like to see done.

At the moment Drools has a user setting "MaintainTMSOption" which can be true or false. It's a small optimisation that when turned off avoids using the equality hashmap that is maintained for all inserted objects.

It would be a much better idea to remove this configuration setting, thus simplifying things for end users and have TMS lazily enabled on demand.

For each object type there is an "ObjectTypeConf" configuration object that is retrieved every time a working memory action, such as insert, is executed. The enabledTMS boolean should be moved there, so there is one per object type, by default it is false.

When a working memory action occurs, like insert, it retrieved the ObjectTypeConf and checks the maintainTms boolean there, instead of the current engine scoped configuration. When a logical insertion occurs and the ObjectTypeConf is retrieved if maintainTms is false it sets the value to true and then iterates the associated ObjectTypeNode memory lazily adding all the objects to the TMS equality map. From then on for that ObjectType all inserted objects are added to that equality map.

With this you now have the advantage of TMS being laziy enabled, so the minor hashmap operation is no longer used and likewise a small memory saving from not populating the map. There is a further advantage that this is now fine grained and when enabled only impacts for that specific object type.

A further enhancement could use a int counter, instead of a boolean. Each logical insertion for that object type increases the counter, each retraction decreases the counter; even if automatically retracted if the truth is broken for that logical assertion. When the counter reaches zero, TMS for that OTN can be disabled. We do not however remove the objects from the equality map, as this would cause "churn" if TMS is continuously enabled and disabled. Instead when TMS is disabled record the current fact counter id. Then if TMS is disabled on a retraction but there is a counter id, we can check that counter id to see if the fact is prior to TMS being disabled and thus would need to be retracted from the equality map.

_______________________________________________
rules-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/rules-dev


_______________________________________________ rules-dev mailing list [hidden email] https://lists.jboss.org/mailman/listinfo/rules-dev


_______________________________________________
rules-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/rules-dev







_______________________________________________
rules-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/rules-dev




--
 - CTO @ http://www.plugtree.com 
 - MyJourney @ http://salaboy.wordpress.com
 - Co-Founder @ http://www.jbug.com.ar
 
 - Salatino "Salaboy" Mauricio -

_______________________________________________
rules-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/rules-dev



_______________________________________________
rules-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/rules-dev
Reply | Threaded
Open this post in threaded view
|

Re: Lazily Enabled Truth Maintenace (Project Propsoal)

Mark Proctor
In reply to this post by leo_gomes
On 23/09/2010 13:44, Leonardo Gomes wrote:
Hi,

I implemented the test cases you suggested, Mark, and started working on the counter. I discussed a bit with Edson yesterday and he clarified the difference between the counter for logically inserted facts and the fact id counter. This part seems to be fine, but I have a problem with the following scenario:

After an initial activation of TMS I put everything in the equality map, then, due to retractions, the counter gets back to zero and I have to disable TMS for that specific type. When a new fact is logically inserted I have to look at the OTN memory and get everything that was inserted there since the last deactivation of TMS, to be be able to re-enable it. To do so I use the fact handle id that I'm supposed to have stored together with the counter. Fine.

Problem: Since the OTN memory is an ObjectHashSet, I will have to iterate through it and filter (there's an ObjectFilter interface that I think I can implement) all the elements that have id greater than the one I stored. This will probably be inefficient and still likely to cause churn.

Possible solutions:

1) Use a TreeMap? ** Don't think so, HashSet retrieval and insertion is O(1), on average, whereas TreeMap is O(log(n));

2) Use a LinkedHashSet (probably a homegrown implementation on top of ObjectHashSet)? ** Would allow me to get the ObjectEntry I want and then call getNext() up to the end, inserting all of them in the equality map;

3) Never disable TMS for a given type once it has been enabled? ** I think that lazily enabling TMS per type might be already enough an optimization, in a sense that the overhead with the calculation of the delta and all this logic to disable/re-enable might not be worthy. Moreover, switching from HashSet to LinkedHashSet may have some disadvantages.
Lets get it working with just lazy enabling to begin with, so for now it can't be disabled once enabled. Once that is working we can look into how we can make it disable and decide if it's worth doing or not.

Mark

What do you think?

Cheers,
Leo.


On Tue, Sep 21, 2010 at 12:09 AM, Mark Proctor <[hidden email]> wrote:
On 20/09/2010 21:58, Leonardo Gomes wrote:
Ok, it took me much longer that I had thought, but I think it's working now.. all tests passing except for ReteooWorkingMemoryTest that does:

        FactHandle fd = workingMemory.insert( string );

        assertEquals( 1,
                      tms.getAssertMap().size() );

Which is a good sign, since tms is now empty as long as you haven't done any logical insert :)

I implemented just the first part of your suggestion, with a boolean. I will try to implement the solution with the counter later this week and maybe do some adjustments before submitting a patch.

Do you have any suggestion in terms of unit tests that you would like to see or just the apparent lack of regressions is enough? Well, I can also think of some tests by taking ReteooWorkingMemoryTest as an example, in case you don't have anything particular in mind.
When you do some insertions cast to get some of the internal data structures and check the equality map is null, maybe check the OT confs are false, do a logical insertion check it can turn on for specific OTNs and not all and that it correctly adds all objects for the OTN. This will be important for the next stage of the work.

Mark


Cheers,
Leo.

 


On Mon, Sep 20, 2010 at 5:14 PM, Mark Proctor <[hidden email]> wrote:
On 20/09/2010 15:53, Leonardo Gomes wrote:
I think I wasn't quite clear in my last email, so let me try to reformulate it:

I also gave it a try to try to do what you suggested here (http://blog.athico.com/2010/09/lazily-enabled-truth-maintenace.html), Mark, and couldn't  make it work due to the following situation:

1) Current code seems to rely on the equality map to know that a logical insert (insertLogical) for an object that has already been regularly inserted (insert) should be ignored;
You would need to lazily maintain an equality map. When the first logical insertion is done we will have to first populate that map from the Object Type Node set of FactHandles.
In the "insert" one of the first lines is:
            ObjectTypeConf typeConf = this.typeConfReg.getObjectTypeConf( this.entryPoint,
                                                                          object );

So we get the ObjectTypeConf before we do anything with the object itself, and we can check if TMS is being maintained for that Object Type.


2) If I apply the modifications that you suggested, from what I understood, things would start to be put in the equality map only when a logical insert is issued;

*Problem*: How would I handle the situation described in item 1, if I don't have anything in the equality map at the moment a logical insert comes in and I have to "lazily activate" TMS?
final Rete source = this.ruleBase.getRete();
ClassObjectType cot = new ClassObjectType( MyClass.class );
Map<ObjectType, ObjectTypeNode> map = source.getObjectTypeNodes( EntryPoint.DEFAULT );
ObjectTypeNode node = map.get( cot );
final ObjectHashSet memory = (ObjectHashSet) workingMemory.getNodeMemory( node );

That "memory" is the set of currently asserted objects for that OTN. You can now iterate that and populate the equality hash map. This is a one off as the flag will be set on the ObjectTypeConf from then on and all objects will be added to the equality map at the point of insertion.

Does that help? All the information is there, you should have to know how to retrieve it :) We don't currently maintain TMS for anything other than the default entry point. Although I think that was a mistake and we will probably move all entrypoints to work with same, but for now you can ignore that and just focus on the default.

Btw this should not be confused with "equality" mode where the equality map has to be maintained by default from the start. Some would argue that a rule engine should only under work and understand on the bases of equality and thus all users must implement hashcode and equals correctly...

--

Moreover, I'm willing to attempt to implement the left and right un-linking and tried to start with this easier task to start to get familiar with drools-core. I already read the article you linked and your article. Would you have any document with an overview of the way drools implements rete?
not really no, Rete has already well documented in a number of papers. The best thing to do is get onto irc and talk directly to edson and I and we can walk you through classes.
http://www.jboss.org/drools/irc.html

Mark


Thanks in advance!

Cheers,
Leo.






On Sat, Sep 18, 2010 at 3:49 AM, Leonardo Gomes <[hidden email]> wrote:
if you look at the AbstractWorkingMemory insert method you'll see one argument is whether it's a logical insertion or not. You'll also see it check the global maintainTMS configuration and also retrieve the ObjectTypeConf. So between those things someone should be able to get it working.

Today, it enters a block where it operates on the equality map and also creates a default handle based on that TMS global option and *regardless* of whether it's a logical insert.

If I'm *not* putting things in the equality map for regular inserts, when a logical insert comes in, but there were already stated inserts, how will I know that? I would create a new handle for the logical insert and do the tms.addLogicalDependency(...), even tough there were regular inserts before and this seems to be a wrong behaviour.

Apparently, today, you can disable TMS and still use logical inserts in your drl, what, I believe, will lead to inconsistent behaviour, but you're at your own risk.

Ideas? I feel that I missed something :)

Cheers,
Leo.

P.S.: I reached the conclusions above based on the fact that LogicalAssertionTest started failing after I did the changes you suggested.

 


On Wed, Sep 15, 2010 at 3:47 PM, Mark Proctor <[hidden email]> wrote:
On 15/09/2010 14:35, Michael Anstis wrote:
Is this in drools-core; or drools-compiler?

Whilst not undertaking to do the work; have a purpose to nose through the code makes understanding easier.
It's all in DroolsCore.

It's a 5 minute hack for me and then 15 minute unit writing test. But I thought I'd write it up in a hope to bring someone else into the fold, we need more help writting the core engine someone else out there must want to work on current edge engine design :)

if you look at the AbstractWorkingMemory insert method you'll see one argument is whether it's a logical insertion or not. You'll also see it check the global maintainTMS configuration and also retrieve the ObjectTypeConf. So between those things someone should be able to get it working.

Mark


On 14 September 2010 16:47, Mark Proctor <[hidden email]> wrote:
Here is another project proposal, this time simpler. I think this one has Wolfgang's name on it ;)

http://blog.athico.com/2010/09/lazily-enabled-truth-maintenace.html

Three weeks ago I posted the project idea for "Left and Right Unlinking". So far there are no takers, so if you are interested let me know :)

In the meantime I tried to think of a simpler enhancement that we would like to see done.

At the moment Drools has a user setting "MaintainTMSOption" which can be true or false. It's a small optimisation that when turned off avoids using the equality hashmap that is maintained for all inserted objects.

It would be a much better idea to remove this configuration setting, thus simplifying things for end users and have TMS lazily enabled on demand.

For each object type there is an "ObjectTypeConf" configuration object that is retrieved every time a working memory action, such as insert, is executed. The enabledTMS boolean should be moved there, so there is one per object type, by default it is false.

When a working memory action occurs, like insert, it retrieved the ObjectTypeConf and checks the maintainTms boolean there, instead of the current engine scoped configuration. When a logical insertion occurs and the ObjectTypeConf is retrieved if maintainTms is false it sets the value to true and then iterates the associated ObjectTypeNode memory lazily adding all the objects to the TMS equality map. From then on for that ObjectType all inserted objects are added to that equality map.

With this you now have the advantage of TMS being laziy enabled, so the minor hashmap operation is no longer used and likewise a small memory saving from not populating the map. There is a further advantage that this is now fine grained and when enabled only impacts for that specific object type.

A further enhancement could use a int counter, instead of a boolean. Each logical insertion for that object type increases the counter, each retraction decreases the counter; even if automatically retracted if the truth is broken for that logical assertion. When the counter reaches zero, TMS for that OTN can be disabled. We do not however remove the objects from the equality map, as this would cause "churn" if TMS is continuously enabled and disabled. Instead when TMS is disabled record the current fact counter id. Then if TMS is disabled on a retraction but there is a counter id, we can check that counter id to see if the fact is prior to TMS being disabled and thus would need to be retracted from the equality map.

_______________________________________________
rules-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/rules-dev


_______________________________________________ rules-dev mailing list [hidden email] https://lists.jboss.org/mailman/listinfo/rules-dev


_______________________________________________
rules-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/rules-dev









_______________________________________________
rules-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/rules-dev
Reply | Threaded
Open this post in threaded view
|

Re: Lazily Enabled Truth Maintenace (Project Propsoal)

Mark Proctor
Work looks good, I've applied the patch.
https://jira.jboss.org/browse/JBRULES-2709

Mark
On 24/09/2010 23:28, Mark Proctor wrote:
On 23/09/2010 13:44, Leonardo Gomes wrote:
Hi,

I implemented the test cases you suggested, Mark, and started working on the counter. I discussed a bit with Edson yesterday and he clarified the difference between the counter for logically inserted facts and the fact id counter. This part seems to be fine, but I have a problem with the following scenario:

After an initial activation of TMS I put everything in the equality map, then, due to retractions, the counter gets back to zero and I have to disable TMS for that specific type. When a new fact is logically inserted I have to look at the OTN memory and get everything that was inserted there since the last deactivation of TMS, to be be able to re-enable it. To do so I use the fact handle id that I'm supposed to have stored together with the counter. Fine.

Problem: Since the OTN memory is an ObjectHashSet, I will have to iterate through it and filter (there's an ObjectFilter interface that I think I can implement) all the elements that have id greater than the one I stored. This will probably be inefficient and still likely to cause churn.

Possible solutions:

1) Use a TreeMap? ** Don't think so, HashSet retrieval and insertion is O(1), on average, whereas TreeMap is O(log(n));

2) Use a LinkedHashSet (probably a homegrown implementation on top of ObjectHashSet)? ** Would allow me to get the ObjectEntry I want and then call getNext() up to the end, inserting all of them in the equality map;

3) Never disable TMS for a given type once it has been enabled? ** I think that lazily enabling TMS per type might be already enough an optimization, in a sense that the overhead with the calculation of the delta and all this logic to disable/re-enable might not be worthy. Moreover, switching from HashSet to LinkedHashSet may have some disadvantages.
Lets get it working with just lazy enabling to begin with, so for now it can't be disabled once enabled. Once that is working we can look into how we can make it disable and decide if it's worth doing or not.

Mark

What do you think?

Cheers,
Leo.


On Tue, Sep 21, 2010 at 12:09 AM, Mark Proctor <[hidden email]> wrote:
On 20/09/2010 21:58, Leonardo Gomes wrote:
Ok, it took me much longer that I had thought, but I think it's working now.. all tests passing except for ReteooWorkingMemoryTest that does:

        FactHandle fd = workingMemory.insert( string );

        assertEquals( 1,
                      tms.getAssertMap().size() );

Which is a good sign, since tms is now empty as long as you haven't done any logical insert :)

I implemented just the first part of your suggestion, with a boolean. I will try to implement the solution with the counter later this week and maybe do some adjustments before submitting a patch.

Do you have any suggestion in terms of unit tests that you would like to see or just the apparent lack of regressions is enough? Well, I can also think of some tests by taking ReteooWorkingMemoryTest as an example, in case you don't have anything particular in mind.
When you do some insertions cast to get some of the internal data structures and check the equality map is null, maybe check the OT confs are false, do a logical insertion check it can turn on for specific OTNs and not all and that it correctly adds all objects for the OTN. This will be important for the next stage of the work.

Mark


Cheers,
Leo.

 


On Mon, Sep 20, 2010 at 5:14 PM, Mark Proctor <[hidden email]> wrote:
On 20/09/2010 15:53, Leonardo Gomes wrote:
I think I wasn't quite clear in my last email, so let me try to reformulate it:

I also gave it a try to try to do what you suggested here (http://blog.athico.com/2010/09/lazily-enabled-truth-maintenace.html), Mark, and couldn't  make it work due to the following situation:

1) Current code seems to rely on the equality map to know that a logical insert (insertLogical) for an object that has already been regularly inserted (insert) should be ignored;
You would need to lazily maintain an equality map. When the first logical insertion is done we will have to first populate that map from the Object Type Node set of FactHandles.
In the "insert" one of the first lines is:
            ObjectTypeConf typeConf = this.typeConfReg.getObjectTypeConf( this.entryPoint,
                                                                          object );

So we get the ObjectTypeConf before we do anything with the object itself, and we can check if TMS is being maintained for that Object Type.


2) If I apply the modifications that you suggested, from what I understood, things would start to be put in the equality map only when a logical insert is issued;

*Problem*: How would I handle the situation described in item 1, if I don't have anything in the equality map at the moment a logical insert comes in and I have to "lazily activate" TMS?
final Rete source = this.ruleBase.getRete();
ClassObjectType cot = new ClassObjectType( MyClass.class );
Map<ObjectType, ObjectTypeNode> map = source.getObjectTypeNodes( EntryPoint.DEFAULT );
ObjectTypeNode node = map.get( cot );
final ObjectHashSet memory = (ObjectHashSet) workingMemory.getNodeMemory( node );

That "memory" is the set of currently asserted objects for that OTN. You can now iterate that and populate the equality hash map. This is a one off as the flag will be set on the ObjectTypeConf from then on and all objects will be added to the equality map at the point of insertion.

Does that help? All the information is there, you should have to know how to retrieve it :) We don't currently maintain TMS for anything other than the default entry point. Although I think that was a mistake and we will probably move all entrypoints to work with same, but for now you can ignore that and just focus on the default.

Btw this should not be confused with "equality" mode where the equality map has to be maintained by default from the start. Some would argue that a rule engine should only under work and understand on the bases of equality and thus all users must implement hashcode and equals correctly...

--

Moreover, I'm willing to attempt to implement the left and right un-linking and tried to start with this easier task to start to get familiar with drools-core. I already read the article you linked and your article. Would you have any document with an overview of the way drools implements rete?
not really no, Rete has already well documented in a number of papers. The best thing to do is get onto irc and talk directly to edson and I and we can walk you through classes.
http://www.jboss.org/drools/irc.html

Mark


Thanks in advance!

Cheers,
Leo.






On Sat, Sep 18, 2010 at 3:49 AM, Leonardo Gomes <[hidden email]> wrote:
if you look at the AbstractWorkingMemory insert method you'll see one argument is whether it's a logical insertion or not. You'll also see it check the global maintainTMS configuration and also retrieve the ObjectTypeConf. So between those things someone should be able to get it working.

Today, it enters a block where it operates on the equality map and also creates a default handle based on that TMS global option and *regardless* of whether it's a logical insert.

If I'm *not* putting things in the equality map for regular inserts, when a logical insert comes in, but there were already stated inserts, how will I know that? I would create a new handle for the logical insert and do the tms.addLogicalDependency(...), even tough there were regular inserts before and this seems to be a wrong behaviour.

Apparently, today, you can disable TMS and still use logical inserts in your drl, what, I believe, will lead to inconsistent behaviour, but you're at your own risk.

Ideas? I feel that I missed something :)

Cheers,
Leo.

P.S.: I reached the conclusions above based on the fact that LogicalAssertionTest started failing after I did the changes you suggested.

 


On Wed, Sep 15, 2010 at 3:47 PM, Mark Proctor <[hidden email]> wrote:
On 15/09/2010 14:35, Michael Anstis wrote:
Is this in drools-core; or drools-compiler?

Whilst not undertaking to do the work; have a purpose to nose through the code makes understanding easier.
It's all in DroolsCore.

It's a 5 minute hack for me and then 15 minute unit writing test. But I thought I'd write it up in a hope to bring someone else into the fold, we need more help writting the core engine someone else out there must want to work on current edge engine design :)

if you look at the AbstractWorkingMemory insert method you'll see one argument is whether it's a logical insertion or not. You'll also see it check the global maintainTMS configuration and also retrieve the ObjectTypeConf. So between those things someone should be able to get it working.

Mark


On 14 September 2010 16:47, Mark Proctor <[hidden email]> wrote:
Here is another project proposal, this time simpler. I think this one has Wolfgang's name on it ;)

http://blog.athico.com/2010/09/lazily-enabled-truth-maintenace.html

Three weeks ago I posted the project idea for "Left and Right Unlinking". So far there are no takers, so if you are interested let me know :)

In the meantime I tried to think of a simpler enhancement that we would like to see done.

At the moment Drools has a user setting "MaintainTMSOption" which can be true or false. It's a small optimisation that when turned off avoids using the equality hashmap that is maintained for all inserted objects.

It would be a much better idea to remove this configuration setting, thus simplifying things for end users and have TMS lazily enabled on demand.

For each object type there is an "ObjectTypeConf" configuration object that is retrieved every time a working memory action, such as insert, is executed. The enabledTMS boolean should be moved there, so there is one per object type, by default it is false.

When a working memory action occurs, like insert, it retrieved the ObjectTypeConf and checks the maintainTms boolean there, instead of the current engine scoped configuration. When a logical insertion occurs and the ObjectTypeConf is retrieved if maintainTms is false it sets the value to true and then iterates the associated ObjectTypeNode memory lazily adding all the objects to the TMS equality map. From then on for that ObjectType all inserted objects are added to that equality map.

With this you now have the advantage of TMS being laziy enabled, so the minor hashmap operation is no longer used and likewise a small memory saving from not populating the map. There is a further advantage that this is now fine grained and when enabled only impacts for that specific object type.

A further enhancement could use a int counter, instead of a boolean. Each logical insertion for that object type increases the counter, each retraction decreases the counter; even if automatically retracted if the truth is broken for that logical assertion. When the counter reaches zero, TMS for that OTN can be disabled. We do not however remove the objects from the equality map, as this would cause "churn" if TMS is continuously enabled and disabled. Instead when TMS is disabled record the current fact counter id. Then if TMS is disabled on a retraction but there is a counter id, we can check that counter id to see if the fact is prior to TMS being disabled and thus would need to be retracted from the equality map.

_______________________________________________
rules-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/rules-dev


_______________________________________________ rules-dev mailing list [hidden email] https://lists.jboss.org/mailman/listinfo/rules-dev


_______________________________________________
rules-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/rules-dev








_______________________________________________ rules-dev mailing list [hidden email] https://lists.jboss.org/mailman/listinfo/rules-dev


_______________________________________________
rules-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/rules-dev
Reply | Threaded
Open this post in threaded view
|

Re: Lazily Enabled Truth Maintenace (Project Propsoal)

Mark Proctor
On 24/09/2010 23:54, Mark Proctor wrote:
Work looks good, I've applied the patch.
https://jira.jboss.org/browse/JBRULES-2709
There was a small bug with this patch which broke during serialisation. ObjectTypeConfs are not marshalled, thus the isTmsEnabled state was being lost. It now re-determines that state during input marshalling.

Mark

Mark
On 24/09/2010 23:28, Mark Proctor wrote:
On 23/09/2010 13:44, Leonardo Gomes wrote:
Hi,

I implemented the test cases you suggested, Mark, and started working on the counter. I discussed a bit with Edson yesterday and he clarified the difference between the counter for logically inserted facts and the fact id counter. This part seems to be fine, but I have a problem with the following scenario:

After an initial activation of TMS I put everything in the equality map, then, due to retractions, the counter gets back to zero and I have to disable TMS for that specific type. When a new fact is logically inserted I have to look at the OTN memory and get everything that was inserted there since the last deactivation of TMS, to be be able to re-enable it. To do so I use the fact handle id that I'm supposed to have stored together with the counter. Fine.

Problem: Since the OTN memory is an ObjectHashSet, I will have to iterate through it and filter (there's an ObjectFilter interface that I think I can implement) all the elements that have id greater than the one I stored. This will probably be inefficient and still likely to cause churn.

Possible solutions:

1) Use a TreeMap? ** Don't think so, HashSet retrieval and insertion is O(1), on average, whereas TreeMap is O(log(n));

2) Use a LinkedHashSet (probably a homegrown implementation on top of ObjectHashSet)? ** Would allow me to get the ObjectEntry I want and then call getNext() up to the end, inserting all of them in the equality map;

3) Never disable TMS for a given type once it has been enabled? ** I think that lazily enabling TMS per type might be already enough an optimization, in a sense that the overhead with the calculation of the delta and all this logic to disable/re-enable might not be worthy. Moreover, switching from HashSet to LinkedHashSet may have some disadvantages.
Lets get it working with just lazy enabling to begin with, so for now it can't be disabled once enabled. Once that is working we can look into how we can make it disable and decide if it's worth doing or not.

Mark

What do you think?

Cheers,
Leo.


On Tue, Sep 21, 2010 at 12:09 AM, Mark Proctor <[hidden email]> wrote:
On 20/09/2010 21:58, Leonardo Gomes wrote:
Ok, it took me much longer that I had thought, but I think it's working now.. all tests passing except for ReteooWorkingMemoryTest that does:

        FactHandle fd = workingMemory.insert( string );

        assertEquals( 1,
                      tms.getAssertMap().size() );

Which is a good sign, since tms is now empty as long as you haven't done any logical insert :)

I implemented just the first part of your suggestion, with a boolean. I will try to implement the solution with the counter later this week and maybe do some adjustments before submitting a patch.

Do you have any suggestion in terms of unit tests that you would like to see or just the apparent lack of regressions is enough? Well, I can also think of some tests by taking ReteooWorkingMemoryTest as an example, in case you don't have anything particular in mind.
When you do some insertions cast to get some of the internal data structures and check the equality map is null, maybe check the OT confs are false, do a logical insertion check it can turn on for specific OTNs and not all and that it correctly adds all objects for the OTN. This will be important for the next stage of the work.

Mark


Cheers,
Leo.

 


On Mon, Sep 20, 2010 at 5:14 PM, Mark Proctor <[hidden email]> wrote:
On 20/09/2010 15:53, Leonardo Gomes wrote:
I think I wasn't quite clear in my last email, so let me try to reformulate it:

I also gave it a try to try to do what you suggested here (http://blog.athico.com/2010/09/lazily-enabled-truth-maintenace.html), Mark, and couldn't  make it work due to the following situation:

1) Current code seems to rely on the equality map to know that a logical insert (insertLogical) for an object that has already been regularly inserted (insert) should be ignored;
You would need to lazily maintain an equality map. When the first logical insertion is done we will have to first populate that map from the Object Type Node set of FactHandles.
In the "insert" one of the first lines is:
            ObjectTypeConf typeConf = this.typeConfReg.getObjectTypeConf( this.entryPoint,
                                                                          object );

So we get the ObjectTypeConf before we do anything with the object itself, and we can check if TMS is being maintained for that Object Type.


2) If I apply the modifications that you suggested, from what I understood, things would start to be put in the equality map only when a logical insert is issued;

*Problem*: How would I handle the situation described in item 1, if I don't have anything in the equality map at the moment a logical insert comes in and I have to "lazily activate" TMS?
final Rete source = this.ruleBase.getRete();
ClassObjectType cot = new ClassObjectType( MyClass.class );
Map<ObjectType, ObjectTypeNode> map = source.getObjectTypeNodes( EntryPoint.DEFAULT );
ObjectTypeNode node = map.get( cot );
final ObjectHashSet memory = (ObjectHashSet) workingMemory.getNodeMemory( node );

That "memory" is the set of currently asserted objects for that OTN. You can now iterate that and populate the equality hash map. This is a one off as the flag will be set on the ObjectTypeConf from then on and all objects will be added to the equality map at the point of insertion.

Does that help? All the information is there, you should have to know how to retrieve it :) We don't currently maintain TMS for anything other than the default entry point. Although I think that was a mistake and we will probably move all entrypoints to work with same, but for now you can ignore that and just focus on the default.

Btw this should not be confused with "equality" mode where the equality map has to be maintained by default from the start. Some would argue that a rule engine should only under work and understand on the bases of equality and thus all users must implement hashcode and equals correctly...

--

Moreover, I'm willing to attempt to implement the left and right un-linking and tried to start with this easier task to start to get familiar with drools-core. I already read the article you linked and your article. Would you have any document with an overview of the way drools implements rete?
not really no, Rete has already well documented in a number of papers. The best thing to do is get onto irc and talk directly to edson and I and we can walk you through classes.
http://www.jboss.org/drools/irc.html

Mark


Thanks in advance!

Cheers,
Leo.






On Sat, Sep 18, 2010 at 3:49 AM, Leonardo Gomes <[hidden email]> wrote:
if you look at the AbstractWorkingMemory insert method you'll see one argument is whether it's a logical insertion or not. You'll also see it check the global maintainTMS configuration and also retrieve the ObjectTypeConf. So between those things someone should be able to get it working.

Today, it enters a block where it operates on the equality map and also creates a default handle based on that TMS global option and *regardless* of whether it's a logical insert.

If I'm *not* putting things in the equality map for regular inserts, when a logical insert comes in, but there were already stated inserts, how will I know that? I would create a new handle for the logical insert and do the tms.addLogicalDependency(...), even tough there were regular inserts before and this seems to be a wrong behaviour.

Apparently, today, you can disable TMS and still use logical inserts in your drl, what, I believe, will lead to inconsistent behaviour, but you're at your own risk.

Ideas? I feel that I missed something :)

Cheers,
Leo.

P.S.: I reached the conclusions above based on the fact that LogicalAssertionTest started failing after I did the changes you suggested.

 


On Wed, Sep 15, 2010 at 3:47 PM, Mark Proctor <[hidden email]> wrote:
On 15/09/2010 14:35, Michael Anstis wrote:
Is this in drools-core; or drools-compiler?

Whilst not undertaking to do the work; have a purpose to nose through the code makes understanding easier.
It's all in DroolsCore.

It's a 5 minute hack for me and then 15 minute unit writing test. But I thought I'd write it up in a hope to bring someone else into the fold, we need more help writting the core engine someone else out there must want to work on current edge engine design :)

if you look at the AbstractWorkingMemory insert method you'll see one argument is whether it's a logical insertion or not. You'll also see it check the global maintainTMS configuration and also retrieve the ObjectTypeConf. So between those things someone should be able to get it working.

Mark


On 14 September 2010 16:47, Mark Proctor <[hidden email]> wrote:
Here is another project proposal, this time simpler. I think this one has Wolfgang's name on it ;)

http://blog.athico.com/2010/09/lazily-enabled-truth-maintenace.html

Three weeks ago I posted the project idea for "Left and Right Unlinking". So far there are no takers, so if you are interested let me know :)

In the meantime I tried to think of a simpler enhancement that we would like to see done.

At the moment Drools has a user setting "MaintainTMSOption" which can be true or false. It's a small optimisation that when turned off avoids using the equality hashmap that is maintained for all inserted objects.

It would be a much better idea to remove this configuration setting, thus simplifying things for end users and have TMS lazily enabled on demand.

For each object type there is an "ObjectTypeConf" configuration object that is retrieved every time a working memory action, such as insert, is executed. The enabledTMS boolean should be moved there, so there is one per object type, by default it is false.

When a working memory action occurs, like insert, it retrieved the ObjectTypeConf and checks the maintainTms boolean there, instead of the current engine scoped configuration. When a logical insertion occurs and the ObjectTypeConf is retrieved if maintainTms is false it sets the value to true and then iterates the associated ObjectTypeNode memory lazily adding all the objects to the TMS equality map. From then on for that ObjectType all inserted objects are added to that equality map.

With this you now have the advantage of TMS being laziy enabled, so the minor hashmap operation is no longer used and likewise a small memory saving from not populating the map. There is a further advantage that this is now fine grained and when enabled only impacts for that specific object type.

A further enhancement could use a int counter, instead of a boolean. Each logical insertion for that object type increases the counter, each retraction decreases the counter; even if automatically retracted if the truth is broken for that logical assertion. When the counter reaches zero, TMS for that OTN can be disabled. We do not however remove the objects from the equality map, as this would cause "churn" if TMS is continuously enabled and disabled. Instead when TMS is disabled record the current fact counter id. Then if TMS is disabled on a retraction but there is a counter id, we can check that counter id to see if the fact is prior to TMS being disabled and thus would need to be retracted from the equality map.

_______________________________________________
rules-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/rules-dev


_______________________________________________ rules-dev mailing list [hidden email] https://lists.jboss.org/mailman/listinfo/rules-dev


_______________________________________________
rules-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/rules-dev








_______________________________________________ rules-dev mailing list [hidden email] https://lists.jboss.org/mailman/listinfo/rules-dev

_______________________________________________ rules-dev mailing list [hidden email] https://lists.jboss.org/mailman/listinfo/rules-dev


_______________________________________________
rules-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/rules-dev
Reply | Threaded
Open this post in threaded view
|

Re: Lazily Enabled Truth Maintenace (Project Propsoal)

leo_gomes
Hi Mark,

I knew that I would end up missing something with the custom serialization.. thanks for the fix.

I'm still willing to work on the unlinking of left and right sides, but I just moved apartments and am struggling to have a proper internet connection at home. It's hard to do anything without bothering you and Edson with questions on IRC :)

I'm also trying to convince my employer to let me work on this during working hours, but I haven't succeeded yet.

Cheers,
Leo.


On Wed, Oct 6, 2010 at 4:07 AM, Mark Proctor <[hidden email]> wrote:
On 24/09/2010 23:54, Mark Proctor wrote:
Work looks good, I've applied the patch.
https://jira.jboss.org/browse/JBRULES-2709
There was a small bug with this patch which broke during serialisation. ObjectTypeConfs are not marshalled, thus the isTmsEnabled state was being lost. It now re-determines that state during input marshalling.

Mark


Mark
On 24/09/2010 23:28, Mark Proctor wrote:
On 23/09/2010 13:44, Leonardo Gomes wrote:
Hi,

I implemented the test cases you suggested, Mark, and started working on the counter. I discussed a bit with Edson yesterday and he clarified the difference between the counter for logically inserted facts and the fact id counter. This part seems to be fine, but I have a problem with the following scenario:

After an initial activation of TMS I put everything in the equality map, then, due to retractions, the counter gets back to zero and I have to disable TMS for that specific type. When a new fact is logically inserted I have to look at the OTN memory and get everything that was inserted there since the last deactivation of TMS, to be be able to re-enable it. To do so I use the fact handle id that I'm supposed to have stored together with the counter. Fine.

Problem: Since the OTN memory is an ObjectHashSet, I will have to iterate through it and filter (there's an ObjectFilter interface that I think I can implement) all the elements that have id greater than the one I stored. This will probably be inefficient and still likely to cause churn.

Possible solutions:

1) Use a TreeMap? ** Don't think so, HashSet retrieval and insertion is O(1), on average, whereas TreeMap is O(log(n));

2) Use a LinkedHashSet (probably a homegrown implementation on top of ObjectHashSet)? ** Would allow me to get the ObjectEntry I want and then call getNext() up to the end, inserting all of them in the equality map;

3) Never disable TMS for a given type once it has been enabled? ** I think that lazily enabling TMS per type might be already enough an optimization, in a sense that the overhead with the calculation of the delta and all this logic to disable/re-enable might not be worthy. Moreover, switching from HashSet to LinkedHashSet may have some disadvantages.
Lets get it working with just lazy enabling to begin with, so for now it can't be disabled once enabled. Once that is working we can look into how we can make it disable and decide if it's worth doing or not.

Mark

What do you think?

Cheers,
Leo.


On Tue, Sep 21, 2010 at 12:09 AM, Mark Proctor <[hidden email]> wrote:
On 20/09/2010 21:58, Leonardo Gomes wrote:
Ok, it took me much longer that I had thought, but I think it's working now.. all tests passing except for ReteooWorkingMemoryTest that does:

        FactHandle fd = workingMemory.insert( string );

        assertEquals( 1,
                      tms.getAssertMap().size() );

Which is a good sign, since tms is now empty as long as you haven't done any logical insert :)

I implemented just the first part of your suggestion, with a boolean. I will try to implement the solution with the counter later this week and maybe do some adjustments before submitting a patch.

Do you have any suggestion in terms of unit tests that you would like to see or just the apparent lack of regressions is enough? Well, I can also think of some tests by taking ReteooWorkingMemoryTest as an example, in case you don't have anything particular in mind.
When you do some insertions cast to get some of the internal data structures and check the equality map is null, maybe check the OT confs are false, do a logical insertion check it can turn on for specific OTNs and not all and that it correctly adds all objects for the OTN. This will be important for the next stage of the work.

Mark


Cheers,
Leo.

 


On Mon, Sep 20, 2010 at 5:14 PM, Mark Proctor <[hidden email]> wrote:
On 20/09/2010 15:53, Leonardo Gomes wrote:
I think I wasn't quite clear in my last email, so let me try to reformulate it:

I also gave it a try to try to do what you suggested here (http://blog.athico.com/2010/09/lazily-enabled-truth-maintenace.html), Mark, and couldn't  make it work due to the following situation:

1) Current code seems to rely on the equality map to know that a logical insert (insertLogical) for an object that has already been regularly inserted (insert) should be ignored;
You would need to lazily maintain an equality map. When the first logical insertion is done we will have to first populate that map from the Object Type Node set of FactHandles.
In the "insert" one of the first lines is:
            ObjectTypeConf typeConf = this.typeConfReg.getObjectTypeConf( this.entryPoint,
                                                                          object );

So we get the ObjectTypeConf before we do anything with the object itself, and we can check if TMS is being maintained for that Object Type.


2) If I apply the modifications that you suggested, from what I understood, things would start to be put in the equality map only when a logical insert is issued;

*Problem*: How would I handle the situation described in item 1, if I don't have anything in the equality map at the moment a logical insert comes in and I have to "lazily activate" TMS?
final Rete source = this.ruleBase.getRete();
ClassObjectType cot = new ClassObjectType( MyClass.class );
Map<ObjectType, ObjectTypeNode> map = source.getObjectTypeNodes( EntryPoint.DEFAULT );
ObjectTypeNode node = map.get( cot );
final ObjectHashSet memory = (ObjectHashSet) workingMemory.getNodeMemory( node );

That "memory" is the set of currently asserted objects for that OTN. You can now iterate that and populate the equality hash map. This is a one off as the flag will be set on the ObjectTypeConf from then on and all objects will be added to the equality map at the point of insertion.

Does that help? All the information is there, you should have to know how to retrieve it :) We don't currently maintain TMS for anything other than the default entry point. Although I think that was a mistake and we will probably move all entrypoints to work with same, but for now you can ignore that and just focus on the default.

Btw this should not be confused with "equality" mode where the equality map has to be maintained by default from the start. Some would argue that a rule engine should only under work and understand on the bases of equality and thus all users must implement hashcode and equals correctly...

--

Moreover, I'm willing to attempt to implement the left and right un-linking and tried to start with this easier task to start to get familiar with drools-core. I already read the article you linked and your article. Would you have any document with an overview of the way drools implements rete?
not really no, Rete has already well documented in a number of papers. The best thing to do is get onto irc and talk directly to edson and I and we can walk you through classes.
http://www.jboss.org/drools/irc.html

Mark


Thanks in advance!

Cheers,
Leo.






On Sat, Sep 18, 2010 at 3:49 AM, Leonardo Gomes <[hidden email]> wrote:
if you look at the AbstractWorkingMemory insert method you'll see one argument is whether it's a logical insertion or not. You'll also see it check the global maintainTMS configuration and also retrieve the ObjectTypeConf. So between those things someone should be able to get it working.

Today, it enters a block where it operates on the equality map and also creates a default handle based on that TMS global option and *regardless* of whether it's a logical insert.

If I'm *not* putting things in the equality map for regular inserts, when a logical insert comes in, but there were already stated inserts, how will I know that? I would create a new handle for the logical insert and do the tms.addLogicalDependency(...), even tough there were regular inserts before and this seems to be a wrong behaviour.

Apparently, today, you can disable TMS and still use logical inserts in your drl, what, I believe, will lead to inconsistent behaviour, but you're at your own risk.

Ideas? I feel that I missed something :)

Cheers,
Leo.

P.S.: I reached the conclusions above based on the fact that LogicalAssertionTest started failing after I did the changes you suggested.

 


On Wed, Sep 15, 2010 at 3:47 PM, Mark Proctor <[hidden email]> wrote:
On 15/09/2010 14:35, Michael Anstis wrote:
Is this in drools-core; or drools-compiler?

Whilst not undertaking to do the work; have a purpose to nose through the code makes understanding easier.
It's all in DroolsCore.

It's a 5 minute hack for me and then 15 minute unit writing test. But I thought I'd write it up in a hope to bring someone else into the fold, we need more help writting the core engine someone else out there must want to work on current edge engine design :)

if you look at the AbstractWorkingMemory insert method you'll see one argument is whether it's a logical insertion or not. You'll also see it check the global maintainTMS configuration and also retrieve the ObjectTypeConf. So between those things someone should be able to get it working.

Mark


On 14 September 2010 16:47, Mark Proctor <[hidden email]> wrote:
Here is another project proposal, this time simpler. I think this one has Wolfgang's name on it ;)

http://blog.athico.com/2010/09/lazily-enabled-truth-maintenace.html

Three weeks ago I posted the project idea for "Left and Right Unlinking". So far there are no takers, so if you are interested let me know :)

In the meantime I tried to think of a simpler enhancement that we would like to see done.

At the moment Drools has a user setting "MaintainTMSOption" which can be true or false. It's a small optimisation that when turned off avoids using the equality hashmap that is maintained for all inserted objects.

It would be a much better idea to remove this configuration setting, thus simplifying things for end users and have TMS lazily enabled on demand.

For each object type there is an "ObjectTypeConf" configuration object that is retrieved every time a working memory action, such as insert, is executed. The enabledTMS boolean should be moved there, so there is one per object type, by default it is false.

When a working memory action occurs, like insert, it retrieved the ObjectTypeConf and checks the maintainTms boolean there, instead of the current engine scoped configuration. When a logical insertion occurs and the ObjectTypeConf is retrieved if maintainTms is false it sets the value to true and then iterates the associated ObjectTypeNode memory lazily adding all the objects to the TMS equality map. From then on for that ObjectType all inserted objects are added to that equality map.

With this you now have the advantage of TMS being laziy enabled, so the minor hashmap operation is no longer used and likewise a small memory saving from not populating the map. There is a further advantage that this is now fine grained and when enabled only impacts for that specific object type.

A further enhancement could use a int counter, instead of a boolean. Each logical insertion for that object type increases the counter, each retraction decreases the counter; even if automatically retracted if the truth is broken for that logical assertion. When the counter reaches zero, TMS for that OTN can be disabled. We do not however remove the objects from the equality map, as this would cause "churn" if TMS is continuously enabled and disabled. Instead when TMS is disabled record the current fact counter id. Then if TMS is disabled on a retraction but there is a counter id, we can check that counter id to see if the fact is prior to TMS being disabled and thus would need to be retracted from the equality map.

_______________________________________________
rules-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/rules-dev


_______________________________________________ rules-dev mailing list [hidden email] https://lists.jboss.org/mailman/listinfo/rules-dev


_______________________________________________
rules-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/rules-dev








_______________________________________________ rules-dev mailing list [hidden email] https://lists.jboss.org/mailman/listinfo/rules-dev

_______________________________________________ rules-dev mailing list [hidden email] https://lists.jboss.org/mailman/listinfo/rules-dev


_______________________________________________
rules-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/rules-dev



_______________________________________________
rules-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/rules-dev