The Beauty of Sponsored Development
As a company that likes to work very closely with our community, we're often given the opportunity to work on some very interesting (and many times very challenging) features. The following is one such feature being sponsored by one of our marquee partners: Read-Write database splitting. They are planning a Liferay deployment that scales to several millions of users.
Because of the dynamic data-intensive nature of this deployment, they've identified that they're likely to to database-bound during peak load. In order to scale database throughput they wanted to us to implement a mechanism within Liferay that would direct transactional (write) and read-only operations to separate data sources.
How was this done?
First, by using one of lesser-known features of the Spring Framework, target sources, we implmented a custom DynamicDataSourceTargetSource to do the job (if you're not familiar with target sources, one of the nifty things you can do is hot-swap a dynamic proxy's target object at runtime). This DynamicDataSourceTargetSource allows us to switch between read and write data sources at the appropriate time.
Secondly, we wanted to take advantage of Spring's transactional boundaries to help us decide when to use the appropriate data source. Before we did that, there were a couple items that needed cleanup and refactoring:
1) Use @Transactional in favor of xml transaction configuration. Annotations are a better fit for transactions because they allow better fine-grained control of transaction rules. Luckily this transition was very easy because all of our services are autogenerated via ServiceBuilder, so very little coding was needed.
2) We created our own @Transactional interface free of Spring dependencies so that it can be used in our plugins environment. This is virtually a mirror of Spring's @Transactional. We decomposed Spring's <tx:annotation-driven /> and replaced it with our own:
<bean id="transactionAdvice" class="org.springframework.transaction.interceptor.TransactionInterceptor">
<property name="transactionManager" ref="liferayTransactionManager" />
<property name="transactionAttributeSource">
<bean class="org.springframework.transaction.annotation.AnnotationTransactionAttributeSource">
<constructor-arg>
<bean class="com.liferay.portal.spring.annotation.PortalTransactionAnnotationParser" />
</constructor-arg>
</bean>
</property>
</bean>
As you can see we had to implement our own PortalTransactionAnnotationParser to read our custom @Transactional annotation.
3) Now that we have the annotations in place, it's very simple to add additional logic to swap data sources. We created a DynamicDataSourceTransactionInterceptor (subclass of Spring's TransactionInterceptor). This interceptor piggybacks spring's transaction boundary logic to help us decide when to use read or write data sources:
<bean id="transactionAdvice" class="com.liferay.portal.dao.jdbc.aop.DynamicDataSourceTransactionInterceptor">
<property name="dynamicDataSourceTargetSource" ref="dynamicDataSourceTargetSource" />
<property name="transactionManager" ref="liferayTransactionManager" />
<property name="transactionAttributeSource">
<bean class="org.springframework.transaction.annotation.AnnotationTransactionAttributeSource">
<constructor-arg>
<bean class="com.liferay.portal.spring.annotation.PortalTransactionAnnotationParser" />
</constructor-arg>
</bean>
</property>
</bean>
Enabling Read-Write Database Splitting
1) Set up your database for master / slave replication. All major databases support at least a master / slave setup. See some sample instructions for the mysql database. Hint: Make sure you create a read-only user for the slave database so that no one can write directly to the slave.
2) Configure your app server / servlet container for each the master and slave databases. Sample ROOT.xml for Tomcat:
<Resource
name="jdbc/LiferayReadPool"
auth="Container"
type="javax.sql.DataSource"
driverClassName="com.mysql.jdbc.Driver"
url="jdbc:mysql://localhost:3308/lportal..."
username="read_only"
password="password"
maxActive="20"
/>
<Resource
name="jdbc/LiferayWritePool"
auth="Container"
type="javax.sql.DataSource"
driverClassName="com.mysql.jdbc.Driver"
url="jdbc:mysql://localhost:3307/lportal..."
username="root"
password="password"
maxActive="20"
/>
3) Set portal-ext.properties to enable the Spring configuration for dynamic data sources:
spring.configs=\
META-INF/base-spring.xml,\
\
META-INF/hibernate-spring.xml,\
META-INF/infrastructure-spring.xml,\
META-INF/management-spring.xml,\
\
META-INF/util-spring.xml,\
\
META-INF/jcr-spring.xml,\
META-INF/messaging-spring.xml,\
META-INF/scheduler-spring.xml,\
META-INF/search-spring.xml,\
\
META-INF/counter-spring.xml,\
META-INF/documentlibrary-spring.xml,\
META-INF/lock-spring.xml,\
META-INF/mail-spring.xml,\
META-INF/portal-spring.xml,\
META-INF/portletcontainer-spring.xml,\
\
META-INF/mirage-spring.xml,\
\
META-INF/ext-spring.xml, \
META-INF/dynamic-data-source-spring.xml
You now have read-write data source setup. Over the next several months we'll post an update with some performance numbers on a read-write configured set up. Enjoy :)
