Method CreateDataFrameFromData

Namespace

SparkTest.NET.Extensions

Assembly

SparkTest.NET.dll

CreateDataFrameFromData<TData>(SparkSession, TData, params TData[])

Creates a DataFrame from a given TData loaded into spark as a JSON file

[Pure]
public static DataFrame CreateDataFrameFromData<TData>(this SparkSession session, TData first, params TData[] rest) where TData : class

Parameters

session SparkSession

session

first TData

first data item

rest TData[]

other data items

Returns

DataFrame

dataframe

Type Parameters

TData

some TData reference type

Examples

Creating a DataFrame from first and rest provided data,

s.CreateDataFrameFromData(
    new { Id = 1 },
    Enumerable.Range(2, 9).Select(i => new { Id = i }).ToArray()
)

Exceptions

NotSupportedException

if the type is not supported in spark

CreateDataFrameFromData<TData>(SparkSession, IEnumerable<TData>)

Creates a DataFrame from a given TData loaded into spark as a JSON file

[Pure]
public static DataFrame CreateDataFrameFromData<TData>(this SparkSession session, IEnumerable<TData> data) where TData : class

Parameters

session SparkSession

session

data IEnumerable<TData>

data

Returns

DataFrame

dataframe

Type Parameters

TData

some TData reference type

Examples

Creating a DataFrame from a provided enumerable,

var list = Enumerable
    .Range(1, 10)
    .Select(i => new { Id = i, Name = $"Some Name {i}" });
var df = s.CreateDataFrameFromData(list);

• Results
• Schema
• Plan

Id	Name
1	Some Name 1
2	Some Name 2
3	Some Name 3
4	Some Name 4
5	Some Name 5
6	Some Name 6
7	Some Name 7
8	Some Name 8
9	Some Name 9
10	Some Name 10

(top = 20)

Exceptions

NotSupportedException

if the type is not supported in spark

InvalidOperationException

if the enumerable is empty