how remotelly get changed users from latest sync date

[Gustáv Pálos, modificado 8 Anos atrás.]

Hello,

I‘m working on Liferay connector to open source identity management product midPoint, but I have a probem.
I use SOAP web services to create/update/delete users over Portal_UserService remotely, it’s working great on direction MidPoint --> Liferay.
But I need to synchronize users also on opposite direction, Liferay --> MidPoint, but I don’t have method to effectively get changed users from latest sync date (only get all users).
I found UserLocalService and his methods to create Dynamic Query when I teoretically can make query example "select User where created>'2015-05-05' OR modified>'2015-05-05' ", but this is only visible locally and not remotely over web services.

There are other solutions than creating additional portlet in Liferay to publish response from dynamicQuery to remote connector over web services?

Thank you for answer and any triks & tips.

Gustav Palos
Identity Engineer

[Andew Jardine, modificado 8 Anos atrás.]

Hi Gustav,

If I understand correctly, you want changes that are made to a user in Liferay to call some kind of endpoint (service?) at MidPoint to keep the two in synch. If this is the case then I would suggest creating a Model Listener hook: https://www.liferay.com/community/wiki/-/wiki/Main/Portal+Hook+Plugins#section-Portal+Hook+Plugins-Model+Listeners (and there are a ton of examples out there on the web)

Essentially, a listener is attached to the User Model object and you wire up the event logic. You would wire up an onAfterUpdate event that would call your MidPoint service with the updated user data.

[Gustáv Pálos, modificado 8 Anos atrás.]

Hi Andew,

thank You for your answer & time, it's an alternative, how Liferay can "PUSH" changed users to MidPoint, but when MidPoint is offline, we lost information about changes.

MidPoint have information about date & time of last synchronization (timestamp - LatestSync). We need a solution to "PULL" changed&created (& ideally deleted) users since this timestamp ("incremental import").
Without this feature, we must do "full import" (getCompanyUsers(...)) each time in cron every minute to synchronize. When we have thousands of users is not good.

[Andew Jardine, modificado 8 Anos atrás.]

Hmmm.... what about this.

1. Register a model listener as previosuly suggested.
2. The model listener will try to communicate changes to MidPoint
3. If the change request fails, then set a flag on the user record (You can use Expando values) to indicate that this record is missing a synch with MidPoint
4. Configure a ScheduledJob (using Liferay's Quartz engine) that will find users where the expando flag is set as "unynched" and then try to synch just those.

Expando details: https://www.liferay.com/community/wiki/-/wiki/Main/Expando
Scheduler: https://liferayazam.wordpress.com/2012/06/10/create-a-scheduler-in-liferay-6/

There are lots of examples out there for both of these topics if you think this will meet your needs.

[Radovan Semancik, modificado 8 Anos atrás.]

Andew,

Thanks for your suggestion. But the problem with this is that it is too complex. We are doing identity management integrations for more than a decade and this type of integration is always problematic. The job may fail due to the bug, it has to be restarted, the flags needs to be managed, they may need manual clearing (e.g. after database restore). The primary problem with this is that it is stateful. State is a problem. We prefer to avoid maintaining extra state if this is possible.

Gustav had noticed that Liferay already has the state that we need. And that's last modification timestamp. What we do in midPoint is that we simply scan for all the records that have been modified since the last scan. This is very simple and very reliable method. We are using similar method to integrate with many systems and it works surprisingly well in practice.

So, all we need is an efficient way to search for accounts (or any other objects) that have modification timestamp later then the specified parameter. Gustav haven't found any way how to do this remotely on a stock Liferay. Is his finding correct or did he overlooked something?

The problem that we have here is that we need to make this search efficiently. Because we will be doing it quite frequently (usually every minute or so). So getting all the accounts and filtering them on the client side is not really an option. Is there any method how to do that search remotely?

--RS

[Andew Jardine, modificado 8 Anos atrás.]

Hi Radovan,

I see. Apologies, I thought that the changes we're being PUSHED to midPoint, not pulled from Liferay. I probably mis-read the post.

User records are also written into a search index. Unfortuantely, the search index is not externally exposed. There are a couple of options I can see.

1. Struts Action Hook for a particular URL that you would call with a date/time parameter. You can make it so that it is onbly available with a secure session and enforce permissions logic in the hook implementation. The hook logic can then use the Liferay Search API to query the search index looking for users with a modified date newer than what you specified. The down side to this would be a case where the DB and the search index are out of date --rare by my experience, but possible.

2. Alternatively, and possibly a better option, create a new service builder portlet but only define a remote interface. This will create for you a SOAP or REST-y (JSON) endpoint that you can call from midpoint. Inside service builder you can define a finder method that would allow you to execute custom SQL. The custom SQL can do a select on the User_ table based on the modified date you provide.

Do you think either or those might be an option for you guys? If not -- I am sure I can dream up something else

[Radovan Semancik, modificado 8 Anos atrás.]

Thanks Andew,

We need to explore these options. I was hoping for some way how to do this in unmodified Liferay. But if that is not possible then the next best thing is the small component in Liferay that you are proposing. Thanks again.

But I still have one curious question. Do you plan to extend the remote interface in the future? I guess that the search capability could be a really nice addition.

--RS

[Andew Jardine, modificado 8 Anos atrás.]

One more thing that perhaps I was missing as part of your architecture.

If you are using an LDAP repo that both these systems are referncing then what you want to do maybe be achievable via configuration. In the past I have done implementations where we had an OpenSSO using SAML auth connected to a OpenDJ directory store. We also connected Liferay to the same directory store using the LDAP configuration. In our case we used a WebAgent to protect the entire portal under a certain domain, but the HTTP Header Param that we passed was used in the import search filter to pull the user record in. So our process was --

1. User tries to access the portal
2. Web Agent intercepts the request and routes the user to the OpenSSO login page.
3. User logs in and OpenSSO manages the whole IdP, SAML token etc. -- login is validated via records in the OpenDJ directory store.
4. The HTTP Param is passed to Liferay and Liferay does an LDAP import of the user record.

... now, we had, in the end, made it so that you could not make changes to the User record in Liferay so that all changes originated in OpenSSO and were passed to OpenDJ. Since the auth manager, really was OpenSSO it didn't matter that the user record in LR might be out of date.

BUT! originally we had it configured to do import and export of user records. Under this model, changes to the Liferay record were EXPORTED back into the OpenDJ directory store meaning that the single reference point for both Liferay and OpenSSO was always in sync.

Does that sound like something you're trying to do?

[Radovan Semancik, modificado 8 Anos atrás.]

Well, yes and no.

The SSO part is a technology of Access Management field. That's somehow independent from identity management (IDM) that we are implementing. We care about the data synchronization. Not so much about authentication.

We have already implemented the ability for Liferay to be a target system (you would perhaps say "slave") for the identity data. So we can replicate data from LDAP or AD or HR system or whatver other gizmo to Liferay. That was easy.

Now we are looking a way how to make Liferay the authoritative source of data. The use case is that in one of our deployments users are registering on the Liferay portal that our customer already has. We need to take that data and create appropriate accounts in LDAP/AD, unix systems and other services that the customer is supposed to access after registration. There is a complex RBAC model for this that is part of the midPoint IDM system. But that's not important. What we need is an efficient service to determine which accounts in Liferay were recently changed. That's enough. MidPoint can then pull every one of the changed account, check what was actually changed, process roles, compute how that affects other accounts and distribute that change accordingly. That's what midPoint usually does.

MidPoint already has a mechanism called reconciliation which lists all the accounts and compares every one of them to check for changes. But the problem is that this a slow process and it has performance impact. So it cannot run all the time. We are expecting to have 10k-50k users in the future. Therefore we are looking to enable what we call "live synchronization" for Liferay. This is an efficient way how to only process accounts that were recently changed. We have this implemented for AD, some LDAP servers and also database tables. Now we want it also for Liferay. And all we need for this to work is an efficient remote way how to get a list of recently changed objects.

[Andew Jardine, modificado 8 Anos atrás.]

I see -- in that case I would probably start by investgating option #2 -- the ervice builder plugin. If you do something like this ...

<service-builder package-path="com.yourcompany.whatever.midpoint">
	<namespace>MidPoint</namespace>
	<entity name="MidPoint" uuid="true" local-service="false" remote-service="true" />
</service-builder>

as your service.xml definition that it will basically do nothing but generate the remote interface for you to fill in with the User lookup details. For that piece you just need to create a search query to lookup the users in the index. If you need help with that piece, just let me know.