用 Python 定義 Schema 并生成 Parquet 文件詳情
發(fā)布日期：2022-01-01 00:33 | 文章來源：源碼中國(guó)
Java和Python實(shí)現(xiàn) Avro 轉(zhuǎn)換成Parquet格式，chema都是在 Avro 中定義的。這里要嘗試的是如何定義Parquet的Schema, 然后據(jù)此填充數(shù)據(jù)并生成Parquet文件。
一、簡(jiǎn)單字段定義
1、定義 Schema 并生成 Parquet 文件
import pandas as pd
import pyarrow as pa
import pyarrow.parquet as pq
# 定義 Schema
schema = pa.schema([
('id', pa.int32()),
('email', pa.string())
])
# 準(zhǔn)備數(shù)據(jù)
ids = pa.array([1, 2], type = pa.int32())
emails = pa.array(['first@example.com', 'second@example.com'], pa.string())
# 生成 Parquet 數(shù)據(jù)
batch = pa.RecordBatch.from_arrays(
[ids, emails],
schema = schema
)
table = pa.Table.from_batches([batch])
# 寫 Parquet 文件 plain.parquet
pq.write_table(table, 'plain.parquet')
import pandas as pd
import pyarrow as pa
import pyarrow . parquet as pq
# 定義 Schema
schema = pa . schema ( [
( 'id' , pa . int32 ( ) ) ,
( 'email' , pa . string ( ) )
] )
# 準(zhǔn)備數(shù)據(jù)
ids = pa . array ( [ 1 , 2 ] , type = pa . int32 ( ) )
emails = pa . array ( [ 'first@example.com' , 'second@example.com' ] , pa . string ( ) )
# 生成 Parquet 數(shù)據(jù)
batch = pa . RecordBatch . from_arrays (
[ ids , emails ] ,
schema = schema
)
table = pa . Table . from_batches ( [ batch ] )
# 寫 Parquet 文件 plain.parquet
pq . write_table ( table , 'plain.parquet' )
2、驗(yàn)證 Parquet 數(shù)據(jù)文件
我們可以用工具 parquet-tools 來查看 plain.parquet 文件的數(shù)據(jù)和 Schema
$ parquet-tools schema plain.parquet message schema {optional int32 id;optional binary email (STRING); } $ parquet-tools cat --json plain.parquet {"id":1,"email":"first@example.com"} {"id":2,"email":"second@example.com"}
沒問題，與我們期望的一致。也可以用 pyarrow 代碼來獲取其中的 Schema 和數(shù)據(jù)
schema = pq.read_schema('plain.parquet')
print(schema)
df = pd.read_parquet('plain.parquet')
print(df.to_json())
schema = pq . read_schema ( 'plain.parquet' )
print ( schema )
df = pd . read_parquet ( 'plain.parquet' )
print ( df . to_json ( ) )
輸出為：
schema = pq.read_schema('plain.parquet')
print(schema)
df = pd.read_parquet('plain.parquet')
print(df.to_json())
schema = pq . read_schema ( 'plain.parquet' )
print ( schema )
df = pd . read_parquet ( 'plain.parquet' )
print ( df . to_json ( ) )
二、含嵌套字段定義
下面的 Schema 定義加入一個(gè)嵌套對(duì)象，在 address 下分 email_address 和 post_address，Schema 定義及生成 Parquet 文件的代碼如下
import pandas as pd
import pyarrow as pa
import pyarrow.parquet as pq
# 內(nèi)部字段
address_fields = [
('email_address', pa.string()),
('post_address', pa.string()),
]
# 定義 Parquet Schema，address 嵌套了 address_fields
schema = pa.schema(j)
# 準(zhǔn)備數(shù)據(jù)
ids = pa.array([1, 2], type = pa.int32())
addresses = pa.array(
[('first@example.com', 'city1'), ('second@example.com', 'city2')],
pa.struct(address_fields)
)
# 生成 Parquet 數(shù)據(jù)
batch = pa.RecordBatch.from_arrays(
[ids, addresses],
schema = schema
)
table = pa.Table.from_batches([batch])
# 寫 Parquet 數(shù)據(jù)到文件
pq.write_table(table, 'nested.parquet')
import pandas as pd
import pyarrow as pa
import pyarrow . parquet as pq
# 內(nèi)部字段
address_fields = [
( 'email_address' , pa . string ( ) ) ,
( 'post_address' , pa . string ( ) ) ,
]
# 定義 Parquet Schema，address 嵌套了 address_fields
schema = pa . schema ( j )
# 準(zhǔn)備數(shù)據(jù)
ids = pa . array ( [ 1 , 2 ] , type = pa . int32 ( ) )
addresses = pa . array (
[ ( 'first@example.com' , 'city1' ) , ( 'second@example.com' , 'city2' ) ] ,
pa . struct ( address_fields )
)
# 生成 Parquet 數(shù)據(jù)
batch = pa . RecordBatch . from_arrays (
[ ids , addresses ] ,
schema = schema
)
table = pa . Table . from_batches ( [ batch ] )
# 寫 Parquet 數(shù)據(jù)到文件
pq . write_table ( table , 'nested.parquet' )
1、驗(yàn)證 Parquet 數(shù)據(jù)文件
同樣用 parquet-tools 來查看下 nested.parquet 文件
$ parquet-tools schema nested.parquet message schema {optional int32 id;optional group address { optional binary email_address (STRING); optional binary post_address (STRING);} } $ parquet-tools cat --json nested.parquet {"id":1,"address":{"email_address":"first@example.com","post_address":"city1"}} {"id":2,"address":{"email_address":"second@example.com","post_address":"city2"}}
用 parquet-tools 看到的 Schama 并沒有 struct 的字樣，但體現(xiàn)了它 address 與下級(jí)屬性的嵌套關(guān)系。
用 pyarrow 代碼來讀取 nested.parquet 文件的 Schema 和數(shù)據(jù)是什么樣子
schema = pq.read_schema("nested.parquet")
print(schema)
df = pd.read_parquet('nested.parquet')
print(df.to_json())
schema = pq . read_schema ( "nested.parquet" )
print ( schema )
df = pd . read_parquet ( 'nested.parquet' )
print ( df . to_json ( ) )
輸出：
id: int32
-- field metadata --
PARQUET:field_id: '1'
address: struct<email_address: string, post_address: string>
child 0, email_address: string
-- field metadata --
PARQUET:field_id: '3'
child 1, post_address: string
-- field metadata --
PARQUET:field_id: '4'
-- field metadata --
PARQUET:field_id: '2'
{"id":{"0":1,"1":2},"address":{"0":{"email_address":"first@example.com","post_address":"city1"},"1":{"email_address":"second@example.com","post_address":"city2"}}}
id : int32
-- field metadata --
PARQUET : field_id : '1'
address : struct & lt ; email_address : string , post_address : string & gt ;
child 0 , email_address : string
-- field metadata --
PARQUET : field_id : '3'
child 1 , post_address : string
-- field metadata --
PARQUET : field_id : '4'
-- field metadata --
PARQUET : field_id : '2'
{ "id" : { "0" : 1 , "1" : 2 } , "address" : { "0" : { "email_address" : "first@example.com" , "post_address" : "city1" } , "1" : { "email_address" : "second@example.com" , "post_address" : "city2" } } }
數(shù)據(jù)當(dāng)然是一樣的，有略微不同的是顯示的 Schema 中, address 標(biāo)識(shí)為 struct<email_address: string, post_address: string> , 明確的表明它是一個(gè) struct 類型，而不是只展示嵌套層次。
到此這篇關(guān)于用 Python 定義 Schema 并生成 Parquet 文件詳情的文章就介紹到這了,更多相關(guān)用 Python 定義 Schema 并生成 Parquet 文件內(nèi)容請(qǐng)搜索本站以前的文章或繼續(xù)瀏覽下面的相關(guān)文章希望大家以后多多支持本站！
版權(quán)聲明：本站文章來源標(biāo)注為YINGSOO的內(nèi)容版權(quán)均為本站所有，歡迎引用、轉(zhuǎn)載，請(qǐng)保持原文完整并注明來源及原文鏈接。禁止復(fù)制或仿造本網(wǎng)站，禁止在非www.sddonglingsh.com所屬的服務(wù)器上建立鏡像，否則將依法追究法律責(zé)任。本站部分內(nèi)容來源于網(wǎng)友推薦、互聯(lián)網(wǎng)收集整理而來，僅供學(xué)習(xí)參考，不代表本站立場(chǎng)，如有內(nèi)容涉嫌侵權(quán)，請(qǐng)聯(lián)系alex-e#qq.com處理。
相關(guān)文章
關(guān)注官方微信