To the platform of Apache Spark, the .NET development can be done now by an open-source package that is known as version 1.0 of .NET for Apache Spark. In the last month of October 2020, Microsoft had officially announced this news. There are various spark and features associated with this newly released version of .NET. The .NET standard 2.0 is targeted in this version by supporting the .NET applications. There is the ability to write Spark SQL as it supports the data frame APIs of Apache Spark. With the help of user-defined functions of .NET, Apache Spark applications can be written.

The users can download this open source project from NuGet and in the GitHub foundation of .NET, it is also available as an operations support system project.  Support in the multiplatform like Linux, macOS, and Windows is offered. In the AWS EMR Spark, and in many other Apache Spark cloud offering .NET for Apache Spark can be used.

Rather than learning Python or Scala for building a big data application is very time taking and a complex process to execute but by using .NET this can be done in a very simpler way. There is a growing interest is observed for the last two years in the market in the .NET community. Keeping the market demand in mind Microsoft had started working on building .NET for Apache Spark by the collaborated team with the members from the .NET team and Azure Data engineering team. In the middle of the year 2019, the public version was declared, and the development process continued with several updates to make it user friendly and prominent. There were twelve different pre-release versions of this .NET for Apache Spark. The release of .NET for Apache Spark is addressed as an initial stage or the beginning by Microsoft. The continuous procedure is going to bring a more upgraded version by consulting with the developers of .NET and the issues that are faced by them are being analyzed to provide the resolution in the next version. There are also some requests by the big data application developers about including the feature of GroupMap. Docker images those are community-contributed “ready to run” and update in the version of .NET is the main target of Microsoft for the future. The facility of releasing jobs from Visual Studio and the collaboration with CD/CI developers is also a priority to support the deployment options.