Add support for MongoDB as JobRepository

[spring-projects-issues]

Ramin Zare opened BATCH-2727 and commented

adding Support for storing Job instances in Mongo instead of JDBC 

---

Issue Links:

• BATCH-2836 Add Support For MongoDB or Api To Be Implemented To Store JobRepository in Any Arbitary Store
  ("is duplicated by")
• BATCH-1596 Support for NoSQL database persistence

[rcardin]

Hey, guys. I and some friends could be interested in implementing this feature. If so, does anyone on the Spring Batch team can help us with some hints?

[fmbenhassine]

@rcardin Thank you for your interest to help! You would need to implement 4 DAO interfaces (JobInstanceDao, JobExecutionDao, StepExecutionDao, ExecutionContextDao) and a factory bean that extends AbstractJobRepositoryFactoryBean. The abstract factory bean already takes care of creating a transactional proxy (that would be based on a MongoTransactionManager) around a SimpleJobRepository with the 4 DAOs. I have created the initial stubs here while evaluating another issue (Point 5 of #3942). For the implementation logic of the DAOs, you can get inspiration from the JDBC -based ones.

You are welcome to contribute if you want. I would love to help if you need support on this!

[rcardin]

Hey @benas. First, thanks for pointing us to your stubs. They're handy. We started to look at the code, and we noticed immediately that the main business classes, such as JobExecution and JobInstance, extend a class called Entity. This class forces us to have a Long for every object.

However, as you know, Mongo doesn't generate ids of type Long, and there is no simple way to generate an autoincremented identifier. I suppose this fact is strictly related to the fact that the business models and the persistent models are collapsed into the same classes.

Do you suggest having a conservative approach, trying to overcome the problem of the generated unique identifier with some trick like these, or to have a more disruptive approach, reviewing the models?

[fmbenhassine]

I think you are hitting an issue similar to #1317. The implications of this choice touch not only the domain model (which is designed around sequences), but also other core parts of the framework, like:

• All APIs related to these entities are built around Long (See input parameters and return types of methods like JobExplorer#getJobExecution, JobOperator#getRunningExecutions, etc)
• The logic of getting the last job/step executions (used in restarts) is based on max(ID)
• All JDBC DAOs use a DataFieldMaxValueIncrementer from Spring Framework which is designed to increment the data store field's maximum value. Incrementing a value implies using a sequence, even if values are of type String for example. The point here is rather about the fact that ordering is required for the previous point to work
• JSR-352 APIs use the long type for IDs. While we are planning to deprecate the support for JSR-352 (in Remove JSR-352 implementation #3894), we need to make sure the version that introduces the required changes here is compatible with the JSR (or delay this feature to a version in which the JSR implementation is removed).

As you can see, the impact of changing the type of Entity ID is quite substantial. While I'm not against reviewing the model, I would like to take time to evaluate such change (or see it in a fork). That said, I think trying to mimic sequences with Mongo's counter documents might work here (using a single document for all the 3 sequences won't work, but using 3 separate documents or even separate collections, one for each sequence might work). Do you want to try the conservative approach with a quick prototype? If we find any blockers, we can consider the disruptive approach. What do you think?

[rcardin]

@benas, thanks for the detailed response. We completely agree with you. We will implement a prototype that mimics sequences using Mongo features.