Multiple schema in one kafka topic

Multiple schema in one kafka topic. 11 ksqldb and cli (seems like much of this might not work in older versions) Example of two tables from two database instances that stream into Kafka topics: Jun 25, 2016 · Yes, the Producer does specify the topic. ADD_TYPE_INFO_HEADERS ), but when I send and receive both messages I get an exception. Jan 8, 2024 · Before consuming messages, we need to subscribe kafkaConsumer to both topics using the subscribe () method: kafkaConsumer. Jan 24, 2019 · 6. Net The Kafka Multitopic Consumer origin reads data from multiple topics in an Apache Kafka cluster. ]+) is to match our topics. Starting with Confluent Schema Registry version 4. transforms=Combine. The origin can use multiple threads to enable parallel processing of data. Both io. You can also use a Kafka output binding to write from your function to a topic. Important Because Debezium uses multiple topics–of which certain limitations may apply–for storing data, Confluent recommends you view Configuring Debezium Topics before you create a database schema history topic. Dec 25, 2020 · Before getting to the strategies a note on how to migrate between schemas. Jul 10, 2020 · Schema Registry now supports schema references in Confluent Platform 5. When you create a stream in ksqlDB based on another you can have it inherit the schema. Dec 20, 2022 · All you need is a Kafka Cluster with the Confluent Schema Registry, 2 KSQL queries/topic running on ksqlDB, and a JDBC Sink Connector running on a Connect cluster. Embedded within the Kafka topic's value contains some information about the source and the schema of the message field. You can use regular expressions (regex) to map the tables to the topic required. bat --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test. The SMT below is useful for more complex transformations of the name. TopicRecord, also set the register automatically to AutomaticRegistrationBehavior. created one consumer with a for loop passing each topic at a time and polling it processing and committing manually. The confluent_kafka_python Schema Registry address accepts a CSV list. ByLogicalTableRouter or org. Sorted by: 12. Award. As Gary mentioned ( here) there is some hint ( JsonSerializer. Note: Most Apache Kafka ™ systems store all messages in the same format and Kafka Connect workers only support a single converter class for key and Have stream processing application (e. Feb 19, 2020 · As usual, Kafka consumers will consume the same messages from the topic and process them for a different purpose. You can use the Apache Kafka trigger in Azure Functions to run your function code in response to messages in Kafka topics. Confluent’s Control Center allows you to manage schemas for Kafka topics. There is no ' leader partition' . Clients define subject strategies to pull ids from the registry. prefix connector configuration property. RegexRouter should do the trick. Oct 8, 2017 · Kafka only guarantees message order for a single topic/partition. Apr 18, 2020 · We have a use-case where we are trying to consumer multiple Kafka topics (AVRO messages) integrating with Schema registry. servers is used to coordinate Schema Registry instances (leader election), and store schema data. 8k 5 69 99. kafka. Use a single topic per entity. The Kafka cluster stores streams of records in categories called topics. bootstrap. The original use case for Kafka was to be able to rebuild a user activity tracking pipeline as a set of real-time publish-subscribe feeds. Other stream processing e. Topics testing docker kafka spring-boot protobuf avro schema-registry gradle docker-compose kafka-streams spring-cloud-stream state-store spring-kafka avro-kafka multiple-event interactive-queries spring-cloud-stream-binder-kafka Jan 30, 2019 · topics. Replication of events in Kafka topics from one cluster to another is the foundation of Confluent’s multi datacenter architecture. Consider customer interactions with an online retailer. For creating a new Kafka Topic, open a separate command prompt window: kafka-topics. Reply. I'm trying to deserialize different JSON payloads from the same Kafka topic. WITH (KAFKA_TOPIC='some_topic_new', VALUE_FORMAT='AVRO') AS. Storing results in Apache Kafka or Database served via Rest API. If this is the first time configuring Kafka on Spring, a good place to start is with our introduction to Apache Kafka with Spring. In the above command, a new topic named baeldung_linux is created with 1 partition and 1 replica set using a Kafka broker on port 9092. edit: this also means you can get messages out of order if your single topic Consumer switches partitions for some reason. This repository contains sample Spring Boot setup for handling multiple event types in single Kafka topic. Consider I have two topics: cust & customers. Nov 21, 2022 · Having single message schema on topic or multiple schemas is always good question and both has some bright sights and drawbacks, you can read more about it in Martin Kleppmann article. Always. 1. Jul 20, 2018 · SteVizzo. Join or union. Typically you have 1:1 topic/schema relationship, but it is possible (and valid) to have multiple schemas per topic in some situations. Number of topics in Kafka is not an issue. When you configure a Kafka Multitopic Consumer, you configure the consumer group name and the Am i right in interpreting the position of the confluent-kafka-dotnet team as "dont use the schema registry in . If used, the key of the Kafka message is often one of the primitive types mentioned above. Unless I am missing something, Transactional Kafka support for producer seems to be very restrictive. Feb 24, 2022 · I need consume messages from one topic that have multiple avro schemas. If you use Jsonschema oneOf or Avro/Protobuf union types in one schema. g. Use Sep 7, 2020 · 1. CREATE STREAM my_stream. you declare in your event schema that the type can either be a user-created or a user-deleted. with different logic for every type of message. So you can try something like. . For example, sharded tables have the same schema. --partitions 1 --replication-factor 1 --bootstrap-server kafka:9092. deleted and so on. Oct 4, 2019 · In this scenario, it would be great to be able to define the Person schema independently of a topic, but to still have schema that use topics as their subjects. Let’s build a books consumer service that listens to the newly The schema type, Compatibility mode, and topic the schema is used by are indicated, along with the schema version and ID. A very simplified version of the message looks something like this: {. Relational databases require a schema. I was reading the Kafka page, the new Producer API is thread-safe and sharing single instance will improve the performance. May 11, 2024 · Discover the power of partitions in Kafka topics and learn how to consume messages from partitioned topics using Spring Boot. 1) for the same: Mar 14, 2019 · If not, then the "slow" topics will slow down your overall consumption, so to achieve smaller delays on "fast" topics go for option 2. But while auto registering the schema it still using the SubjectNameStrategy. Aug 18, 2018 · When working with a combination of Confluent Schema Registry + Apache Kafka, you may notice that pushing messages with different Avro schemas to one topic was not possible. Each message is appended to the end of a topic, and consumers can read messages from one or more topics. 2. readStream. Jul 2, 2020 · With that you can configure all of them to use the same topic and belong to the same group. A topic is a category or feed name to which messages are published. In your case you will have 3 leaders, each of ᅠ ᅠ ᅠ ᅠ ᅠ ᅠ ᅠ ᅠ ᅠ ᅠ ᅠ ᅠ ᅠ ᅠ ᅠ ᅠ ᅠ ᅠ ᅠ ᅠ ᅠ ᅠ ᅠ ᅠ Select Download Format Kafka Topic Multiple Avro Schemas Download Kafka Topic Multiple Avro Schemas PDF Download Kafka Topic Multiple Avro Schemas DOC ᅠ Manifests itself in conflict with the day is creating data format has the development. The tool reads from a source cluster and writes to a destination cluster. Nov 2, 2018 · Support for multiple schemas in one topic for JAVA avro serializer was added in pull request: Jul 28, 2021 · You just need to ensure that a single set of your service instances are assigned the same value of consumer group ID for that config. Target Aug 2, 2021 · I am trying to add multiple schemas to the same subject in the schema registry, so I have set ValueSubjectNameStrategy to SubjectNameStrategy. However, it appears that I'm getting an exception while the consumer tries to read the message. io/blog/put-several-event-types-kafka-topic/ answered Jan 21, 2019 at 15:59. They can be replicated in order to make the data fault-tolerant and highly available. Oct 20, 2023 · For example, if there are three schemas for a subject that change in order 1, 2, and 3 then FULL_TRANSITIVE compatibility ensures that consumers using the new schema 3 can process data written by Mar 15, 2019 · How to manage multiple schemas in a topic ? In many scenarios you will end up producing multiple event types into a single topic, how to efficiently manage it ? Schema Registry is exactly meant for that, to manage & govern data models across topics. asList(topic1,topic2), ConsumerRebalanceListener obj) Consumer has the topic info and we can commit using consumer. type=io. If you have multiple nodes producing to Kafka the order is obviously not guaranteed. Or, you can setup a reverse proxy, single address, for your Registry cluster. format takes a single value, and defaults to using the topic name itself. Kafka comes with a tool for mirroring data between Kafka clusters. Dec 4, 2023 · In Kafka, producers are responsible for publishing messages to topics. net if you have multiple types per topic" if i have misunderstood sorry, but i was hoping we could use the schema registry and get all the schema evolutionary benefits etc. This isn't a problem and saves storage on the brokers by not storing duplicate information. I tried to use a GenericRecord Type to deserialize the message without pass the avro schema, but the serialization not working well because return a string with invalid json format. Sep 6, 2020 · To read from Kafka for streaming queries, we can use function SparkSession. Kafka topics are partitioned, which distributes data across multiple brokers for scalability. , in an e-commerce application, there could be an ‘orders’ topic. apache. Here we have a use case where a stream processing app that produces two different message types: Dec 29, 2018 · I have one topic from which I can receive different types of jsons. servers is also used to specify security protocols that Schema Registry uses to connect to Kafka. You're maybe mixing up replicas and partitions. Regarding ordering it is a bit more complicated in a real-life distributed system. transforms. You'll need to use a plain loop to register to distinct Registries, however, it's not good practice to have Nov 26, 2018 · The DataStax Connector allows for mapping a single topic to multiple tables for a single connector instance. subscribe(Arrays. Let’s publish one message on each of the topics: Dec 29, 2021 · I think what @Oxy means is there is only one constant for schema name. 30. asList( "card-payments", "bank-transfers" )); We’re ready now to test our configuration. Lookup topic. Nov 6, 2023 · Now, let’s create a Kafka topic: $ docker-compose exec kafka kafka-topics. Sep 7, 2018 · KSQL, the SQL streaming engine for Apache Kafka®, puts the power of stream processing into the hands of anyone who knows SQL. Dec 12, 2017 · And in my online course on Apache Avro, the Confluent Schema Registry and Kafka REST proxy, I go over these concepts in great depth alongside many hands-on examples. The Avro subject could essentially be anything. Method used is consumer created common and in for loop I am subscribing one topic at a time and processing data. Project Setup. Jun 19, 2019 · Multiple message types per Kafka topic. The customer searches through the site and clicks on various items before deciding on a final purchase. Even if you have a single node but async producer the order is not guaranteed. Building real-time streaming applications that transform or react to the streams of data. I would like to understand if it is feasible (and it makes sense) to publish/receive different types on the same topics with Spring Cloud Stream. And I have a consumer that must consume all type of message. The other questions asked here, guided me to a first attempt, but I was not able to get it running. confluent. Replication can be done with Confluent Replicator or using the open source Kafka MirrorMaker. You are correct that there are two solutions for sending messages with multiple schemas on the same topic: either a top-level union in the avro schema, or multiple schemas per topic. The connector writes schema change events to a Kafka topic named <topicPrefix>, where topicPrefix is the namespace specified in the topic. In the example stocks_topic, the key is a basic string and the value is regular JSON. Topics also retain events even after Dec 1, 2021 · Create multiple consumers for same topic in kafka 15 How to Consume from specific TopicPartitionOffset with Confluent. registry. One strategy for achieving that is to use union types, i. regex: will source data from every topic that matches this pattern as you can see I added RegexRouter transformation , to dynamically extract the topic name before sinking to MySQL, the pattern I used: ([^. A rough formula for picking the number of partitions is based on throughput. However you have a leader broker that handle a partition. Sep 2, 2018 · The SMT provided by Debezium enables you to rewrite the topic and the key according to a regular expression and a replacement pattern, configurable per instance of Debezium. Jan 12, 2022 · In the further steps, you will be seeing how to create Kafka Topics and configure them for efficient message transfer. topic. Jul 25, 2022 · In practice this means you must use the same topic for all those events, which means this topic will contain events with different schemas. When consumption from a kafka topic with 10 Jan 8, 2024 · Kafka uses topics to store and categorize these events, e. info("event={}", event); But in this case all messages come to Aug 24, 2016 · As some one already commented you should not push different type of schema's to single topic. connect. Knowing the internals provides context when troubleshooting or trying to understand why Kafka behaves the way it does, and also helps you set configuration parameters. Robin Moffatt. send(new ProducerRecord<byte[],byte[]>(topic, partition, key1, value1) , callback); The more partitions there are in a Kafka cluster, the higher the throughput one can achieve. May 3, 2022 · To do this I can change the Avro schema file, regenerate POJOs using Maven schema plugin, and then if I'll send a message to the first topic with a KafkaTemplate, the schema will be updated and the new field will be visible on the topic. Topic strategy. Assuming you're using the Confluent Schema Registry, then each topic is a unique subject, but the schemas can indeed be shared since matching schemas would generate matching ids. Jun 15, 2020 · and if we have multiple schema definitions for various records in our topics we have a problem where we need to define one producer per schema. The JDBC Sink connector relies on the JDBC API and database-specific drivers to write data from a Kafka topic into a table on the The full name of the Kafka topic where the connector will store the database schema history. 5, and this blog post presents an alternative means of putting several event types in the same topic using schema references, discussing the advantages and disadvantages of this approach. And if you have a sync producer - your code blocks on each send. Use an Avro union. 4. Nov 20, 2016 · 4,079 7 37 61. Explanation with an example. We can subscribe for multiple topic using following API : consumer. SchemaRegistry and Confluent. When streaming the data from Kafka I want to apply the Spark schema to the Kafka message based on the topic name. url", "ur1,url2"); How can I use the same producer to send messages to these two different topics even though they have different schema. Apr 15, 2020 · Kafka Connect's Single Message Transform capabilities. For more information, see https://www. Jul 15, 2018 · Schemas of employee and department are registered in schema registry. Aug 21, 2016 · I have multiple producers that can send multiple type of event to one kafka topic. As a rule of thumb, if you care about latency, you Apr 30, 2020 · I have a Spark Structured Streaming job which is streaming data from multiple Kafka topics based on a subscribePattern and for every Kafka topic I have a Spark schema. Nov 4, 2021 · In these cases, using Kafka, it's often not desirable to publish the messages over multiple topics. In this section, we'll take a deep dive into Kafka Internals and learn how Kafka handles topic storage with segments and indexes. Spark can subscribe to one or more topics and wildcards can be used to match with multiple topic names similarly as the batch query example provided above. One of the key features that make Kafka powerful is its ability to handle schema evolution and data compatibility challenges efficiently. In some scenarios, events defined by different schemas can be published to the same Apache Kafka ® topic. config. 4 ) , Confluent Kafka (Library version: 5. No SMT ships with Apache Kafka that does this but there are prototypes out there that could do this. Nov 26, 2018 · Ingest multiple topics and write to different tables using a single connector instance. I tried to add additional bean names but that didn't work. 0, you can do it and I will explain to you how. setProperty("schema. In a logical table, there are multiple physical tables that all have the same schema. For information about supported versions, see Supported Systems and Versions. There's an example of the latter in this post here: Jul 8, 2019 · Above code work fine for me as single topic but i have to listen messages from multiple topic and created multiple AvroGenerated files but i stuck in configuration as confiration need multipe avro type objects. Use DynamicMessage and access the message by constructing the object fields using FieldDescriptors. It’s fun to use for exploring data in Kafka topics, but its real power comes in building stream processing applications. If you have multiple entities in a single topic, it’s difficult to keep track of which messages belong to which entity. If your service B has 10 instances, you specify a common kafka consumer group ID for all the 10 instances, say serviceBConsumerGroup. ([^. When you decided to have multiple events on single topic, starting from rest api and next by Kafka producer and consumer you can use the same approach of . 3. ByLogicalTableRouter. e. Given that choice, we can't have atomic transaction that spans multiple output sink topics. For information on setup and configuration details, see Apache Kafka bindings for Azure Functions overview. Chronological order or out-of-order. Replicator can be used for replication of topic data as well as migrating schemas in Schema Registry. Schema Registry and Avro Serialization are excellent tools for achieving the changes in schema very efficiently. commitSync() by creating OffsetAndMetadata object as follows. ConsumerRecords<String, String> records = consumer. So how would you configure multiple? @OneCricketeer As I understand the docs if iI want to produce from for example a kstream application, I configure my kafka client with some: AWSSchemaRegistryConstants, there is no obvious way to set multiple schema names there. df = spark \. @KafkaListener(topics = "test", containerFactory = "kafkaListenerContainerFactory") logger. If you are wondering why you have to send messages with different schema on a single topic, this is best explained in this article : Multiple Event Types. Jul 24, 2022 · 3 min read. Image Source. Something like: Jan 20, 2022 · Per the docs, table. my_table. Apr 12, 2018 · Today I have built a Spark Structured Streaming application which consumes a single Kafka topic which contain JSON messages. Kafka server addresses and topic names are required. These cases are handled smoothly in Kafka and Schema Registry, but until now, processing them with ksqlDB was quite complex. 31. I using a c# lib Confluent. symbol text, Apr 29, 2020 · When we produce an Avro record to a Kafka topic, our producer needs to encode the Avro schema into it and serialzse it into a byte array. You can by default use the same Avro schema in multiple topics, but you cannot use multiple Avro schemas for one topic. readStream \. poll Aug 23, 2016 · I am just exploring Kafka, currently i am using One producer and One topic to produce messages and it is consumed by one Consumer. very simple. Now I have only one consumer app in which I should be able to apply different schema dynamically (by inspecting May 9, 2016 · 26. So, if you use something other than TopicNameStategy, the default, then you'll naturally end up with multiple typed records sent into one topic. Combine. It seems that its trying reading from the topic and trying to convert to all the types that are reading from the topic. That's still only Jul 18, 2021 · In My application Kafka topics are dedicated to a domain (can't change that) and multiple different types of events (1 Event = 1 Avro schema message) related to that domain being produced by different micro-services in that one topic. name. In order to do this you will need to maintain multiple Avro schema versions either in the same schema using unions or in the same or different libraries using namespaces. Disclaimer: this assumes v0. Kafka in . Jan 18, 2018 · But since each topic in Kafka has at least one partition, if you have n topics, you inevitably have at least n partitions. When you subscribe to multiple topics with a single Consumer, that Consumer is assigned a topic/partition pair for each requested topic. The Managing multiple event types in a single topic with Schema Registry. But different event types can be related. Oct 29, 2021 · Messages to both topics are only validated against url2 instead of validated topic1 against url1 and topic2 against url2. Data will be read from topics in the source cluster and written to a topic with the same name Sep 12, 2023 · Kafka is a popular choice for building data pipelines, real-time analytics, and event-driven applications. First a few concepts: Kafka is run as a cluster on one or more servers that can span multiple datacenters. In the DataStax keyspace stocks_keyspace, create three different tables that optimized with different schemas. called "obj3flag" within a JSON structure) in the Key of the Kafka message that tells your structured streaming job if the obj_3 is existing in the Kafka Value and then choose either the one or the other schema to parse the json string. The default subject naming strategy TopicNameStrategy does not allow for this because it couples the schema subject name to the topic name. On the other hand, when we consume the Avro record, our consumer needs to deserialize the byte array and decode it using the Avro schema into a text or object that our human eyes can read. You can use some nomenclature to manage them. urls? Am I setting this config incorrectly? Aug 27, 2018 · we have one consumer group and three topics, all three topics are of different schema . This means site activity (page views, searches, or other actions users may take) is published to central topics with one topic per activity type. The issue is that if I'll send a message with UserAction to the second topic, it would still refer to the Aug 4, 2019 · 4. You measure the throughout that you can achieve on a single partition for production (call it p) and consumption (call it c ). Both options will load balance into "one" Registry (since there's one master server). my_table and db_shard2. Type: list A logical table is a common use case for routing records for multiple physical tables to one topic. Thanks to foreachBatch we can easily separate the processing and, therefore, use 2 different schemas for our inconsistent datasets and process them independently inside one common application. "source": "Application A", Oct 31, 2020 · I'm using Confluent Schema Registry with Avro and I'd like to use multiple schemas in one Kafka Topic. transforms. Apr 26, 2023 · Should clarify that topics don't have schemas. Apr 5, 2020 · The second solution I found answers the question about the content of Dataset if it's processed with foreachBatch and multiple topics subscription is used. Each record consists of a key, a value, and a timestamp. format("kafka") \. Sep 23, 2020 · Now, we'd like to make it easy for us to join these topics and stream the results into a new topic which could be consumed by multiple services. regex=(. Using the Kafka sink connector, is it possible to separate data coming for each entity based on schema and write to respective tables Example, employee data coming to topic should go to employee table and department data to department table. A while ago, Jun Rao wrote a blog post explaining the cost of having many partitions (end-to-end latency, file descriptors, memory overhead, recovery time after a failure). sh --create --topic baeldung_linux. *) Jul 24, 2022 · Kafka provides a mechanism for handling messages with multiple schemas on a given topic. When dealing with different schemas, consider using a schema registry or schema management tool. Jul 24, 2022. 1. . Kafka Streams; Edit: I mean can I define two different jdbc sink to different oracle tables from one kafka topic . If you just process a message and store it into the same database, I think you just need to create a single consumer. using Kafka Streams) between producer and consumer, which splits up the messages from a shared topic with both kinds of messages to two separate topics (again using a message header), each consumed by one of the groups. Messages that the connector sends to the schema change topic contain a payload, and, optionally, also contain the schema of the change event message. ]+)\. Jan 21, 2019 · 1 Answer. Does it mean i can use single Producer to publish messages to multiple topics? Join multiple topics. Jul 21, 2022 · Handling multi-schema Protobuf/Avro topics in ksqlDB. Typically, a schema will be used by only one topic. Nov 13, 2022 · By following these best practices, you can design Kafka topics that are scalable, reliable, and easy to maintain. This can lead to confusion and data loss. Published in Hadoop. Challenges. If you want to replicate data from one cluster to another then there is one kafka tool called MirrorMaker. Yes, each topic can be consumed by multiple sinks. Note that it won't the same schema though, but the definition will be the same. [event-name] and then I extracted only group 2 (event-name). We are using Spark Structured streaming ( Spark version : 2. Based on the Schema Registry Naming Strategies documentation, it looks like clients can be configured to use either topics or records as the subject of schema across all topics/schema Jul 8, 2020 · Before these newer subject-name strategies were introduced, there were two options for storing multiple event types in the same topic: Disable subject-schema constraints by setting the compatibility level of a subject to NONE and allowing any schema to be saved in the subject, regardless of compatibility. May 12, 2021 · use a flag (e. producer. Jan 8, 2024 · In this article, we’ll learn how to configure multiple listeners for the same Kafka topic by looking at a practical example. ·. debezium. With Apache Kafka, it's typical to place different events in their own topic. When you say you have a topic with 3 partitions, it means your records will be dispatched amongs them according to the record key ( or dispatcher algo) . Both Avro and Protobuf offer this feature. In this article, we will explore how to use the Kafka Producer API to publish messages to multiple topics. Although if you have only one partition in the topic, the concurrency won't happen: a single partition is processed in a single thread. Mar 12, 2015 · Therefore, in general, the more partitions there are in a Kafka cluster, the higher the throughput one can achieve. Feb 17, 2023 · Next steps. The exception to this is if you have multiple clusters in the same environment, which have topics with the same name and that name matches a schema subject name based on The Kafka cluster containing the bootstrap servers specified in kafkastore. created, account. WITH (KAFKA_TOPIC='some_topic', VALUE_FORMAT='AVRO'); CREATE STREAM my_stream_new. commitAsync or consumer. By continually streaming messages from one Kafka topic to another, applying transformations Oct 14, 2017 · Sorted by: 5. If you run several such topologies in parallel on different nodes (for high availability or high throughput), then having one consumer group (option 1) will result in more consumers to coordinate within it. Avro uses the topic name as schema name (subject). Yes, use Single Message Transforms, per the link you identified. A logical table might consist of two or more sharded tables: db_shard1. When sending a message to a topic t, the Avro schema for the key and the value will be automatically registered in Schema Registry under the subject t-key and t-value, respectively, if the compatibility test passes. You can however change the Avro schema subject to something unique, such as the canonical name of the class you Nov 2, 2023 · Hi @Joe1912 , Certainly! Handling multiple Kafka topics with different schemas in a single job can be achieved using various approaches. Sep 18, 2018 · Can anyone provide me a small example in spring boot kafka where we can consume multiple topics in one single listener class. When Kafka security is enabled, kafkastore. The DataStax Connector allows for mapping multiple topics to multiple tables in a single connector instance. Edit: courtesy of @OneCricketeer, you can also just use table. regex CID1122. format=kafka_${topic}_V1. The Kafka coordinator will distribute partitions to your consumers. Kafka for make my consumer. Suppose I have a Kafka topic named account with several message types (each one with a different Avro schema), like account. 29 5. Dec 4, 2022 · 0. Click through to see how this works out. like "topic1", "topic1_json", "topic1_modification". Kafka Synchronizing Database: synchronize one Kafka topic containing multiple tables or multiple topics containing one table each into one Paimon database. wq hs ne os pj vq rt tx mj yp