Apache Spark 2.0 Now Available
Apache Spark 2.0, released in late July, is now available on the Altiscale Data Cloud. As one of the most active open-source Big Data projects under development, Spark is an integral component of the Big Data ecosystem and is used by a majority of Altiscale customers in areas such as interactive SQL and data transformation. The addition of Spark 2.0 to Altiscale’s Spark-as-a-Service offering provides improved performance, expanded SQL support, and streamlined programming APIs.
New in Spark 2.0
Spark 2.0 delivers substantial improvements over its predecessor, Spark 1.6. Through more intelligent code generation that emits optimized code at runtime and eliminates unnecessary overhead, Spark 2.0 targets a 10x performance gain. With the inclusion of a new SQL parser in Spark 2.0, Altiscale customers who use Spark SQL will benefit from Spark’s newly added SQL:2003 support, which reduces the need for users to modify existing applications that require SQL:2003 features. Spark 2.0 also introduces Structured Streaming to unify batch and streaming analytics. Structured Streaming simplifies the programming model for users, as they can now process static data and unbounded data streams using a single high-level API.
Concurrent Support for Multiple Versions of Spark
With Spark’s frequent releases—a new version approximately every three months—organizations can benefit from the rapid pace of community-driven innovation in Spark technology, but they are challenged to ensure the stability of existing applications built on prior versions. As the only provider of Spark-as-a-Service whose management extends to the application level, Altiscale automatically deploys and runs multiple recent versions of Spark (2.0, 1.6, 1.5) so that customers can migrate to newer versions of Spark at their own pace.
Altiscale customers should have Spark 2.0 deployed currently or in their next maintenance window. Customers should contact Altiscale support if they require Spark 2.0 in their environments before their next maintenance. More information on Spark 2.0 on Altiscale can be found at https://documentation.altiscale.com/spark-2-0-with-altiscale.