CloudCanalProcess Homepage, Documentation and Download- CloudCanal SDK Data Processor- News Delivery

CloudCanal Data Process is an open source data processor for CloudCanal SDK, and the native official SDK only has some documentation.

This project is a custom code processor initiated by officials and jointly promoted by the community. This project brings together CloudCanal data processing plug-ins to achieve the goal of data custom transformation.

Plug-in description

  • wide-table : Plug-in for wide table data processing, mainly including combined processing of fact tables and single-dimensional tables
  • data-transform : General data conversion plug-ins, such as changing operations, adding additional fields, cleaning and backfilling data
  • data-gather : Data aggregation plug-in, real-time aggregation of sub-database sub-table, vertical split, and remote data
  • data-compare : Data comparison plug-in, business reconciliation according to changes in source data
  • business-alert : Business alarm plug-in, make corresponding alarms according to data change trends

Instructions for use

  • Install CloudCanal and create a data migration synchronization task
  • Perform adaptive transformation of the CloudCanalProcessor implementation class (such as: WideTableProcessorV2_simple) that needs to be used
  • Modify it in src/main/resources/META-INF/cloudcanal/ under the subproject to the class that needs to be used
  • under subproject mvn -Dtest -DfailIfNoTests=false -Dmaven.javadoc.skip=true -Dmaven.compile.fork=true clean package Pack
  • CloudCanal console creation task (Reference case article), and uploadSubproject jar package under target (eg: wide-table-1.0.0-SNAPSHOT.jar)


Data Migration Synchronization Architecture

related resources

#CloudCanalProcess #Homepage #Documentation #Download #CloudCanal #SDK #Data #Processor #News Delivery

Leave a Reply

Your email address will not be published. Required fields are marked *