joget workflow is using hibernate. by right any database supported by hibernate should work but it seem like not officially supported by the joget team.
u can try to add the sybase jdbc driver to the tomcat lib directory and setting up the db by refer to Custom Database Configuration.
