{"id":61167,"date":"2025-02-02T09:10:12","date_gmt":"2025-02-02T05:40:12","guid":{"rendered":"https:\/\/afaghhosting.net\/blog\/?p=61167"},"modified":"2025-02-02T09:10:12","modified_gmt":"2025-02-02T05:40:12","slug":"%d9%86%da%af%d8%a7%d9%87%db%8c-%d8%a8%d9%87-%d9%82%d8%a7%d9%84%d8%a8%d9%87%d8%a7%db%8c-%d8%ac%d8%af%db%8c%d8%af-%d8%b0%d8%ae%db%8c%d8%b1%d9%87-%d8%af%d8%a7%d8%af%d9%87%d9%87%d8%a7","status":"publish","type":"post","link":"https:\/\/afaghhosting.net\/blog\/%d9%86%da%af%d8%a7%d9%87%db%8c-%d8%a8%d9%87-%d9%82%d8%a7%d9%84%d8%a8%d9%87%d8%a7%db%8c-%d8%ac%d8%af%db%8c%d8%af-%d8%b0%d8%ae%db%8c%d8%b1%d9%87-%d8%af%d8%a7%d8%af%d9%87%d9%87%d8%a7\/","title":{"rendered":"\u0646\u06af\u0627\u0647\u06cc \u0628\u0647 \u0642\u0627\u0644\u0628\u200c\u0647\u0627\u06cc \u062c\u062f\u06cc\u062f \u0630\u062e\u06cc\u0631\u0647 \u062f\u0627\u062f\u0647\u200c\u0647\u0627"},"content":{"rendered":"

\u0628\u0627 \u06af\u0633\u062a\u0631\u0634 \u0645\u0641\u0627\u0647\u06cc\u0645\u06cc \u0645\u0627\u0646\u0646\u062f LakeHouse \u0647\u0627 \u0648 \u0627\u0633\u062a\u0627\u0646\u062f\u0627\u0631\u062f\u0647\u0627\u06cc\u06cc \u0645\u0627\u0646\u0646\u062f IceBerg \u0648 \u062a\u0633\u0647\u06cc\u0644 \u0627\u0645\u06a9\u0627\u0646 \u0627\u062c\u0631\u0627\u06cc \u06a9\u0648\u0626\u0631\u06cc \u0628\u0631 \u0631\u0648\u06cc \u0641\u0627\u06cc\u0644\u200c\u0647\u0627\u06cc \u062f\u0627\u062f\u0647 \u067e\u0631\u062f\u0627\u0632\u0634 \u0646\u0634\u062f\u0647 (\u062e\u0627\u0645 )\u060c \u0642\u0627\u0644\u0628 \u0630\u062e\u06cc\u0631\u0647 Parquet \u0648 \u062a\u0627 \u062d\u062f\u0648\u062f\u06cc \u0647\u0645 ORC \u0628\u0647 \u06cc\u06a9 de facto \u0627\u0633\u062a\u0627\u0646\u062f\u0627\u0631\u062f \u062f\u0631 \u0627\u06cc\u0646 \u062d\u0648\u0632\u0647 \u062a\u0628\u062f\u06cc\u0644 \u0634\u062f\u0647 \u0627\u0633\u062a \u0648 \u062f\u0631 \u0686\u0646\u062f \u0633\u0627\u0644 \u0627\u062e\u06cc\u0631\u060c \u0631\u0634\u062f \u0627\u0633\u062a\u0641\u0627\u062f\u0647 \u0627\u0632 \u0622\u0646\u0647\u0627 \u0631\u0627 \u0634\u0627\u0647\u062f \u0628\u0648\u062f\u0647\u200c\u0627\u06cc\u0645.
\n\u0628\u0627 \u0627\u06cc\u0646 \u0648\u062c\u0648\u062f \u0628\u0647 \u0646\u0638\u0631 \u0645\u06cc\u200c\u0631\u0633\u062f \u062f\u0631 \u0645\u0631\u062d\u0644\u0647 \u06af\u0630\u0627\u0631 \u0627\u0632 \u0627\u06cc\u0646 \u0642\u0627\u0644\u0628\u200c\u0647\u0627\u06cc \u06a9\u0644\u0627\u0633\u06cc\u06a9 \u0630\u062e\u06cc\u0631\u0647 \u0633\u062a\u0648\u0646\u06cc \u062f\u0627\u062f\u0647\u200c\u0647\u0627 \u0628\u0647 \u0642\u0627\u0644\u0628\u200c\u0647\u0627\u06cc \u0630\u062e\u06cc\u0631\u0647 \u062f\u0627\u062f\u0647\u200c\u0647\u0627\u06cc \u062e\u0627\u0645 \u0628\u0627 \u0636\u0631\u06cc\u0628 \u0641\u0634\u0631\u062f\u06af\u06cc \u0628\u0627\u0644\u0627\u062a\u0631 \u0648 \u0628\u0647\u06cc\u0646\u06af\u06cc \u0628\u0633\u06cc\u0627\u0631 \u0628\u06cc\u0634\u062a\u0631 \u062f\u0631 \u067e\u0631\u062f\u0627\u0632\u0634 \u0648 \u067e\u06cc\u0645\u0627\u06cc\u0634 \u0647\u0633\u062a\u06cc\u0645 .
\n\u062a\u0639\u062f\u0627\u062f\u06cc \u0627\u0632\u06cc\u0646 \u0642\u0627\u0644\u0628\u200c\u0647\u0627\u06cc \u062c\u062f\u06cc\u062f \u0630\u062e\u06cc\u0631\u0647 \u062f\u0627\u062f\u0647\u200c\u0647\u0627 \u0628\u0647 \u0635\u0648\u0631\u062a \u062e\u0627\u0645 (\u0628\u062f\u0648\u0646 \u0646\u06cc\u0627\u0632 \u0628\u0647 \u062f\u06cc\u062a\u0627\u0628\u06cc\u0633 \u0628\u0631\u0627\u06cc \u0630\u062e\u06cc\u0631\u0647 \u0627\u06cc\u0646 \u0627\u0637\u0644\u0627\u0639\u0627\u062a) \u062f\u0631 \u0645\u0642\u0627\u0644\u0647 \u0632\u06cc\u0631 \u0645\u0639\u0631\u0641\u06cc \u0648 \u0628\u0631\u0631\u0633\u06cc \u0634\u062f\u0647\u200c\u0627\u0646\u062f.<\/p>\n

\n

\"Dipankar<\/p>\n

\n

Dipankar Mazumdar, M.Sc \"\ud83e\udd51\" on LinkedIn: #parquet #bigdata #dataengineering #softwareengineering<\/p>\n

\u201cMake Apache Parquet 10-100x Faster \"\ud83d\ude80\"\u201d That\u2019s one of the motivations! There is no denying in the fact that the #Parquet file format has been instrumental\u2026<\/p>\n

https:\/\/www.linkedin.com\/posts\/dipankar-mazumdar_parquet-bigdata-dataengineering-activity-7253095572268613632-Wk2r<\/p>\n<\/div>\n<\/div>\n

\n

Advice<\/p>\n

\u0646\u06a9\u062a\u0647 \u0645\u0647\u0645 \u062f\u0631 \u0645\u0648\u0631\u062f \u0627\u06cc\u0646 \u0645\u0648\u0636\u0648\u0639 \u0627\u06cc\u0646 \u0627\u0633\u062a \u06a9\u0647 \u0647\u0631 \u0686\u0642\u062f\u0631 \u0642\u0627\u0644\u0628\u200c\u0647\u0627\u06cc \u0645\u0648\u062b\u0631\u062a\u0631 \u0648 \u0641\u0634\u0631\u062f\u0647\u200c\u062a\u0631\u06cc \u0628\u0631\u0627\u06cc \u0630\u062e\u06cc\u0631\u0647 \u062e\u0627\u0645 \u062f\u0627\u062f\u0647\u200c \u0627\u06cc\u062c\u0627\u062f \u0634\u0648\u062f\u060c \u0631\u0648\u0627\u062c LakeHouse \u0647\u0627 \u06cc\u0627 \u0633\u0627\u0645\u0627\u0646\u0647\u200c\u0647\u0627\u06cc \u062a\u062d\u0644\u06cc\u0644\u06cc \u0645\u0628\u062a\u0646\u06cc \u0628\u0631 \u0641\u0627\u06cc\u0644\u200c\u0647\u0627\u06cc \u062e\u0627\u0645 \u062f\u06cc\u062a\u0627 \u0633\u0631\u0639\u062a \u0628\u06cc\u0634\u062a\u0631\u06cc \u062e\u0648\u0627\u0647\u062f \u06af\u0631\u0641\u062a.<\/p>\n<\/div>\n

\u062f\u0631 \u0646\u0638\u0631 \u0628\u06af\u06cc\u0631\u06cc\u062f :
\n\u2013 \u0633\u0627\u0645\u0627\u0646\u0647\u200c\u0647\u0627\u06cc \u0630\u062e\u06cc\u0631\u0647 \u0633\u0627\u0632\u06cc \u0645\u0627\u0646\u0646\u062f s3 \u0628\u0633\u06cc\u0627\u0631 \u0631\u0627\u06cc\u062c \u0634\u062f\u0647\u200c\u0627\u0646\u062f<\/strong> \u0648 \u0647\u0632\u06cc\u0646\u0647 \u0627\u0633\u062a\u0641\u0627\u062f\u0647 \u0627\u0632 \u0622\u0646\u0647\u0627 \u0647\u0645 \u0628\u0633\u06cc\u0627\u0631 \u06a9\u0627\u0647\u0634 \u06cc\u0627\u0641\u062a\u0647 \u0627\u0633\u062a.\u200c
\n\u2013 \u06a9\u062a\u0627\u0628\u062e\u0627\u0646\u0647\u200c\u0647\u0627\u06cc \u067e\u0631\u062f\u0627\u0632\u0634 \u062f\u0627\u062f\u0647\u060c \u0628\u0633\u06cc\u0627\u0631 \u062d\u0631\u0641\u0647\u200c\u0627\u06cc \u062a\u0631 \u0648 \u0645\u0648\u062b\u0631\u062a\u0631 \u0634\u062f\u0647\u200c\u0627\u0646\u062f<\/strong> (\u0645\u062b\u0644\u0627 polars \u062f\u0631 \u0645\u0642\u0627\u0628\u0644 pandas)
\n\u2013 \u0627\u0633\u062a\u0627\u0646\u062f\u0627\u0631\u062f\u0647\u0627\u06cc\u06cc \u0628\u0631\u0627\u06cc \u0633\u0627\u062e\u062a\u0627\u0631\u062f\u0647\u06cc \u0628\u0647 \u0641\u0627\u06cc\u0644\u200c\u0647\u0627\u06cc \u062e\u0627\u0645 \u0627\u06cc\u062c\u0627\u062f \u0634\u062f\u0647\u200c\u0627\u0646\u062f \u06a9\u0647 \u062d\u062a\u06cc \u0627\u0645\u06a9\u0627\u0646 \u0627\u062c\u0631\u0627\u06cc \u062a\u0631\u0627\u06a9\u0646\u0634\u200c\u0647\u0627\u06cc ACID \u0631\u0627 \u0647\u0645 \u0631\u0648\u06cc \u062f\u0627\u062f\u0647\u200c\u0647\u0627\u06cc \u062e\u0627\u0645 \u0641\u0631\u0627\u0647\u0645 \u0645\u06cc\u200c\u06a9\u0646\u0646\u062f(Apache Iceberg)<\/strong>
\n\u2013 \u06a9\u0627\u062a\u0627\u0644\u0648\u06af\u200c\u0647\u0627\u06cc\u06cc \u0645\u0627\u0646\u0646\u062f Polaris \u060c \u0645\u0633\u0623\u0644\u0647 \u0633\u0637\u062d \u062f\u0633\u062a\u0631\u0633\u06cc \u0648 \u0645\u0633\u0627\u06cc\u0644 \u0627\u0645\u0646\u06cc\u062a\u06cc \u0645\u0631\u062a\u0628\u0637 \u0628\u0627 \u0627\u06cc\u0646 \u0641\u0627\u06cc\u0644\u0647\u0627\u06cc \u062e\u0627\u0645 \u0631\u0627 \u0628\u0631\u0637\u0631\u0641 \u06a9\u0631\u062f\u0647\u200c\u0627\u0646\u062f<\/strong>.
\n\u2013 \u0627\u0628\u0632\u0627\u0631\u0647\u0627\u06cc \u062f\u0645\u200c\u062f\u0633\u062a\u06cc \u0645\u0627\u0646\u0646\u062f DuckDB \u0628\u0631\u0627\u06cc \u06a9\u0627\u0631 \u0628\u0627 \u0627\u06cc\u0646 \u0627\u0633\u062a\u0627\u0646\u062f\u0627\u0631\u062f\u0647\u0627\u060c \u0627\u0631\u062a\u0642\u0627 \u06cc\u0627\u0641\u062a\u0647\u200c\u0627\u0646\u062f<\/strong> \u2026
\n\u2013 \u062e\u06cc\u0644\u06cc \u0627\u0632 \u0645\u0646\u0627\u0628\u0639 \u062f\u0627\u062f\u0647\u200c\u0627\u06cc \u0645\u0627 \u0632\u06cc\u0631 \u06cc\u06a9 \u062a\u0631\u0627\u0628\u0627\u06cc\u062a \u0647\u0633\u062a\u0646\u062f.<\/strong>(\u067e\u0633\u062a \u0627\u062e\u06cc\u0631 \u0639\u0644\u06cc\u0631\u0636\u0627 \u0635\u0627\u062f\u0642\u06cc \u0631\u0627 \u062f\u0631 \u0627\u06cc\u0646 \u0632\u0645\u06cc\u0646\u0647 \u0627\u0632 \u062f\u0633\u062a \u0646\u062f\u0647\u06cc\u062f)
\nhttps:\/\/lnkd.in\/d7W467Fb<\/a>
\n\u0628\u0647 \u0686\u0647 \u0646\u062a\u06cc\u062c\u0647\u200c\u0627\u06cc \u0645\u06cc\u200c\u0631\u0633\u06cc\u062f \u061f \u0622\u06cc\u0627 \u0638\u0647\u0648\u0631 \u0628\u0627\u0632\u06cc\u06af\u0631\u0627\u0646 \u062c\u062f\u06cc\u062f \u0648 \u0631\u0648\u0627\u062c \u0627\u06cc\u0646 \u0642\u0627\u0644\u0628\u200c\u0647\u0627\u06cc \u062d\u0631\u0641\u0647\u200c\u0627\u06cc \u0630\u062e\u06cc\u0631\u0647 \u062f\u0627\u062f\u0647\u200c\u0647\u0627 \u062f\u0631 \u062f\u0646\u06cc\u0627\u06cc \u0645\u0647\u0646\u062f\u0633\u06cc \u062f\u0627\u062f\u0647 \u06a9\u0647 \u0647\u0645 \u0633\u0631\u0639\u062a \u067e\u0631\u062f\u0627\u0632\u0634 \u062f\u06cc\u062a\u0627 \u0631\u0627 \u062a\u0636\u0645\u06cc\u0646 \u062e\u0648\u0627\u0647\u0646\u062f \u06a9\u0631\u062f \u0648 \u0647\u0645 \u0646\u06cc\u0627\u0632 \u0628\u0647 \u0627\u0633\u062a\u0641\u0627\u062f\u0647 \u0627\u0632 \u062f\u06cc\u062a\u0627\u0628\u06cc\u0633 \u0631\u0627 \u0628\u0631\u0627\u06cc \u0628\u0633\u06cc\u0627\u0631\u06cc \u0627\u0632 \u062f\u0627\u062f\u0647\u200c\u0647\u0627\u06cc \u063a\u06cc\u0631\u062d\u06cc\u0627\u062a\u06cc \u0633\u0627\u0645\u0627\u0646\u0647\u200c\u0647\u0627\u060c \u0627\u0632 \u0628\u06cc\u0646 \u062e\u0648\u0627\u0647\u0646\u062f \u0628\u0631\u062f\u060c \u062f\u0648\u0631 \u0627\u0632 \u0627\u0646\u062a\u0638\u0627\u0631 \u0646\u062e\u0648\u0627\u0647\u062f \u0628\u0648\u062f\u061f<\/p>\n

\u0646\u06a9\u0627\u062a \u0627\u0635\u0644\u06cc \u0645\u0642\u0627\u0644\u0647 \u0641\u0648\u0642 :<\/p>\n

Now, in the past year or so, there has been a huge effort in bringing other file formats.<\/p>\n

\"\u2705\" Some of these formats take inspiration from Parquet at some level but are targeted towards specific workloads (say unstructured data \u2013 machine learning)<\/p>\n

\"\u2705\" Formats like <\/a>BTRBlocks<\/a> uses a set of lightweight encoding schemes<\/strong>, achieving fast & efficient decompression & high compression ratios (GitHub Address<\/a>).<\/p>\n

\"\u2705\" Lance by LanceDB<\/a> use cases\u2019 are more targeted towards ML<\/strong> (multi modal). Claims 100x faster than Parquet. (check out this blog post<\/a>)<\/p>\n

\"\u2705\" Nimble by Meta<\/a> is a new columnar file format for large datasets<\/strong>. It is meant to be a replacement for file formats such as Parquet, ORC. Suited for ML use cases (feature store).<\/p>\n

\"\u2705\" Vortex<\/a> is another one that claims to provide faster random access reads (100-200x faster) and scans (2-10x faster)<\/strong>, while preserving approximately the same compression ratio and write throughput as Parquet with ZSTD. (Vortex\u2019s default compression strategy is based on the\u00a0BtrBlocks<\/a>\u00a0paper.)<\/p>\n","protected":false},"excerpt":{"rendered":"

\u0628\u0627 \u06af\u0633\u062a\u0631\u0634 \u0645\u0641\u0627\u0647\u06cc\u0645\u06cc \u0645\u0627\u0646\u0646\u062f LakeHouse \u0647\u0627 \u0648 \u0627\u0633\u062a\u0627\u0646\u062f\u0627\u0631\u062f\u0647\u0627\u06cc\u06cc \u0645\u0627\u0646\u0646\u062f IceBerg \u0648 \u062a\u0633\u0647\u06cc\u0644 \u0627\u0645\u06a9\u0627\u0646 \u0627\u062c\u0631\u0627\u06cc \u06a9\u0648\u0626\u0631\u06cc \u0628\u0631 \u0631\u0648\u06cc \u0641\u0627\u06cc\u0644\u200c\u0647\u0627\u06cc \u062f\u0627\u062f\u0647 \u067e\u0631\u062f\u0627\u0632\u0634 \u0646\u0634\u062f\u0647 (\u062e\u0627\u0645 )\u060c \u0642\u0627\u0644\u0628 \u0630\u062e\u06cc\u0631\u0647 Parquet \u0648 \u062a\u0627 \u062d\u062f\u0648\u062f\u06cc \u0647\u0645 ORC \u0628\u0647 \u06cc\u06a9 de facto \u0627\u0633\u062a\u0627\u0646\u062f\u0627\u0631\u062f \u062f\u0631 \u0627\u06cc\u0646 \u062d\u0648\u0632\u0647 \u062a\u0628\u062f\u06cc\u0644 \u0634\u062f\u0647 \u0627\u0633\u062a \u0648 \u062f\u0631 \u0686\u0646\u062f \u0633\u0627\u0644 \u0627\u062e\u06cc\u0631\u060c \u0631\u0634\u062f \u0627\u0633\u062a\u0641\u0627\u062f\u0647 \u0627\u0632 \u0622\u0646\u0647\u0627 \u0631\u0627 \u0634\u0627\u0647\u062f \u0628\u0648\u062f\u0647\u200c\u0627\u06cc\u0645. \u0628\u0627 …<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[35],"tags":[],"class_list":["post-61167","post","type-post","status-publish","format-standard","hentry","category-bigdata"],"_links":{"self":[{"href":"https:\/\/afaghhosting.net\/blog\/wp-json\/wp\/v2\/posts\/61167","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/afaghhosting.net\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/afaghhosting.net\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/afaghhosting.net\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/afaghhosting.net\/blog\/wp-json\/wp\/v2\/comments?post=61167"}],"version-history":[{"count":1,"href":"https:\/\/afaghhosting.net\/blog\/wp-json\/wp\/v2\/posts\/61167\/revisions"}],"predecessor-version":[{"id":61208,"href":"https:\/\/afaghhosting.net\/blog\/wp-json\/wp\/v2\/posts\/61167\/revisions\/61208"}],"wp:attachment":[{"href":"https:\/\/afaghhosting.net\/blog\/wp-json\/wp\/v2\/media?parent=61167"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/afaghhosting.net\/blog\/wp-json\/wp\/v2\/categories?post=61167"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/afaghhosting.net\/blog\/wp-json\/wp\/v2\/tags?post=61167"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}