Exploring Spark and ML.NET with F#

This repo contains code shown in On .NET Live Show - Exploring Spark and ML.NET with F#

Scenario

Train and deploy a machine learning model to predict the score given to a restaurant after inspection using features such as the number of critical violations, violation type, and a few others.

Prerequisites

• .NET 5.0 SDK or later
• Configure .NET for Apache Spark
• VS Code(Optional)
• Ionide extension(Optional)
• .NET Interactive Notebooks extension(Optional)

Project Assets

• data - Location for original dataset.
• Web API - F# Saturn Web API to host InspectionModel.zip regression model.
  • Domain.fs - Contains the schema of the model input and output.
  • Program.fs - Web API Entrypoint
  • InspectionModel.zip - Regression model to predict the score of a restaurant inspection
• data-exploration.dib - .NET Interactive Notebook to explore and visualize data.
• data.fsx - F# Interactive script to prepare data for training.
• machine-learning.fsx - F# Interactive script to train regression model.
• submit.cmd - Console script to start .NET for Apache Spark in Debug mode.

About the data

The dataset represents the NYC restaurant inspection results. For more information, see DOHMH New York City Restaurant Inspection Results.

Run application

Prepare data

.NET for Apache Spark is used to prepare the raw data for training. To prep data, start Apache Spark.

1. Start Apache Spark in debug mode. Note that this is a script for the Windows CMD prompt. You might need to modify the ^ characters a bit to work on PowerShell ` or Linux \.

   submit.cmd

2. Use .NET CLI and F# Interactive to run data.fsx.

   dotnet fsi data.fsx
   

This will prepare the data, create an output directory prepdata for the results, and save the results to .csv files.

Remember to stop Apache Spark once data.fsx is done preparing the data. Ctrl + C

Train model

ML.NET is used to train a regression model to predict the inspection score. To train your model, use the .NET CLI and F# Interactive to run machine-learning.fsx.

dotnet fsi machine-learning.fsx

This script trains your model and saved the serialized version of it to a file called InspectionModel.zip.

Run Web API

Saturn is used to create an HTTP endpoint to make predictions with the InspectionModel.zip model. Use the .NET CLI to start the Web API

Navigate to the webapi directory and start your application

dotnet run

Make predictions

Once the Web API starts, send an HTTP POST request to http://localhost:5000/predict with the following JSON body:

{
    "Camis":"41720083",
    "Boro":"Manhattan",
    "InspectionType":"Manhattan,Cycle Inspection / Re-inspection",
    "InspectionScore": 11.0,
    "Violations":"04H,09C,10F",
    "CriticalViolations":1.0,
    "TotalViolations":1.0
}

If the request is handled successfully, you should get a response similar to the following:

{
  "InspectionScore": 11,
  "Score": 7.8174763
}

Resources

• F# Documentation
• .NET for Apache Spark Documentation
• ML.NET Documentation
• ML.NET F# Samples
• .NET Interactive