[jira] [Created] (KYLIN-2618) Still dictionary cannot go over 2GB given ForestDict

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

[jira] [Created] (KYLIN-2618) Still dictionary cannot go over 2GB given ForestDict

JIRA jira@apache.org
liyang created KYLIN-2618:
-----------------------------

             Summary: Still dictionary cannot go over 2GB given ForestDict
                 Key: KYLIN-2618
                 URL: https://issues.apache.org/jira/browse/KYLIN-2618
             Project: Kylin
          Issue Type: Improvement
            Reporter: liyang


This is because dictionary persists through ResourceStore API and that goes through a copying process based on byte array. The byte array is limited to 2GB at most.

The solution ideally should be storing dictionary/snapshot directly to HDFS.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)