Kafka - Avro 포맷

Primitive Types

null: 값 없음
boolean: 이진 값
int: 32비트 부호 있는 정수
long: 64비트 부호 있는 정수
float: 단정밀도(32비트) IEEE 754 부동소수점 숫자
double: 배정밀도(64비트) IEEE 754 부동소수점 숫자
bytes: 8비트 부호 없는 바이트의 순서
string: 유니코드 문자 시퀀스

 

예시

avro type	json type	example
null	null	null
boolean	boolean	true
int,long	integer	1
float,double	number	1.1
bytes	string	"\u00FF"
string	string	"foo"
record	object	{"a": 1}
enum	string	"FOO"
array	array	[1]
map	object	{"a": 1}
fixed	string	"\u00ff"

 

 

{
  "type": "record",
  "name": "LongList",
  "aliases": ["LinkedLongs"],                      // old name for this
  "fields" : [
    {"name": "value", "type": "long"},             // each element has a long
    {"name": "next", "type": ["null", "LongList"]} // optional next element
  ]
}

 

 

 

 

스키마 자동 생성

 

---

 

• Json
• Avro
• Protobuf

Json/Avro 포맷의 경우 schema와 payload로 구성

Schema는 해당 레코드의 schema 구성을, payload는 해당 레코드의 값을 가짐.
Connector 별로 Json 포맷은 조금씩 다를수 있지만 전반적으로 대부분 비슷.
Json Schema의 경우 레코드 별로 Schema를 가 지고 있으므로 메시지 용량이 커짐. 
이의 개선을 위해 Avro Format과 Schema Registry를 이용하여 Schema 정보의 중복 생성 제거.

 

---

 

 

참고자료

 

json 포맷 vs Avro 포맷 비교

https://www.confluent.io/blog/schemas-contracts-compatibility/

 

Avro 포맷

https://avro.apache.org/docs/1.11.1/specification/_print/

Specification

 

AWS Glue - schema registery

https://docs.aws.amazon.com/ko_kr/glue/latest/dg/schema-registry-gs.html

Schema Registry 시작하기 - AWS Glue

 

 

kafka connect 프로퍼티 정리

https://godekdls.github.io/Apache%20Kafka/kafka-connect/

Kafka Connect

 

avro 변환 

https://konbert.com 

Convert Files, Databases and Data Online - Konbert

 

 

aws glue git

https://github.com/awslabs/aws-glue-schema-registry/blob/master/integration-tests/kafka-connect-avro-configs/file-sink-standalone.properties

 

 

https://github.com/awslabs/aws-glue-schema-registry/blob/master/README.md

 

 

https://p-bear.tistory.com/48

Apache Avro 란

 

https://louisdev.tistory.com/63

[Apache Avro] Avro 구조와 사용법