7.3.2. Hardware System - Neural Networks#

Design Challenge#

To build your interactive Tic-Tac-Toe exhibit, you must design the physical hardware system. The exhibit must receive enough power to operate, but not so much power that the components burn out. You will use machine learning to predict the resistance of the wiring based on its length.

Task 4: System Block Diagram#

The first step is to engage in problem decomposition by breaking the hardware system down into smaller pieces.

Instructions#

  • Work with your team to draft a block diagram of your physical system.

  • Your diagram must include:

    • four (4) screens,

    • four (4) ways for museum-goers to interact with the screens,

    • four (4) computer chips to run the algorithm,

    • a power source, and

    • wiring.

  • Save this diagram to submit as a PDF.

Task 5: Machine Learning Using a Neural Network#

Lab bench with a yellow multimeter wired to a power supply, an alligator clip on a wire, and a meter stick.

Fig. 7.2 A laboratory setup for measuring the electrical resistance of various lengths of copper wire. A hand adjusts the alligator clip along the yellow wire while the meter stick below measures the length of wire in the circuit.#

The total electrical resistance of your system depends directly on the length of wire used. You conducted a physical test on the specific copper wire planned for the exhibit, gathering the measurements shown in Table 7.1. You will use this experimental data to train your machine learning models.

Instructions#

Open the Neural Network Resistance Excel File.

Setting Up the Neural Network#

To train the neural network, we need to begin by estimating the value for the bias and weight. Open the “Neural Network” tab.

  1. Estimate the initial parameters by entering a bias of \(0.5\) in cell A18 and a weight of \(1\) in cell B18.

  2. Set the learning rate to \(1\) in cell C14.

  3. Input the lengths of wire from Table 7.1 into cells D18 through D27 using absolute cell referencing (from cells B3:B12 on the “Data” tab).

Table 7.1 Wire Resistance Data#

Length (cm)

Resistance (Ohms)

5

2.14

10

3.38

15

3.86

20

4.58

25

8.68

30

8.96

35

10.06

40

12.99

45

14.43

50

14.37

  1. Compute the weighted sum (\(z_1\)) in your spreadsheet using the formula \(z_1 = w_1 \cdot x_1 + b_1 \)

  2. Because you are predicting a real number, set the activation (\(a_1\)) equal to \(z_1\). Cells F18:F27 should match E18:E27.

  3. Compute the cost function in your spreadsheet using the formula \(C_{w,b} = (y_i - a_{1,i})^2\). Then calculate the sum of the cost function for all training data in cell G28.

Training via Gradient Descent#

Next, you must calculate the gradient for both the weight and the bias to determine how they need to change. To calculate the gradient, you would need to compute the derivative of the cost function with respect to both \(w\) and \(b\) using partial derivatives. These derivatives have been calculated for you below.

Next, your team must calculate the gradient for both the weight and the bias to determine how they need to change. To calculate the gradient, you would need to compute the derivative of the cost function with respect to both \(w\) and \(b\) using partial derivatives. These derivatives have been calculated for you below.

(7.2)#\[\Delta b = -k\frac{\delta C}{\delta b}\]
(7.3)#\[\frac{\delta C}{\delta b} = \frac{\delta C}{\delta a} \cdot \frac{\delta a}{\delta z} \cdot \frac{\delta z}{\delta b}\]

(7.4)#\[\Delta w = -k\frac{\delta C}{\delta w}\]
(7.5)#\[\frac{\delta C}{\delta w} = \frac{\delta C}{\delta a} \cdot \frac{\delta a}{\delta z} \cdot \frac{\delta z}{\delta w}\]

Where:

(7.6)#\[\frac{\delta C}{\delta a} = 2(a-y)\]
(7.7)#\[\frac{\delta a}{\delta z} = \frac{e^z}{(1 + e^z)^2}\]
(7.8)#\[\frac{\delta z}{\delta b} = 1\]
(7.9)#\[\frac{\delta z}{\delta w} = x\]

  1. In cells H18-H27, compute \(\frac{\delta C}{\delta a}\) using absolute and relative cell referencing.

    Example: H18 = 2*(F18-$C$3)

  2. In cells I18-I27, compute \(\frac{\delta a}{\delta z}\) using absolute and relative cell referencing.

    Example: I18 = EXP(E18)/(1+EXP(E18))^2

  3. Compute \(\frac{\delta z}{\delta b}\) in cells J18:J27 and \(\frac{\delta z}{\delta w}\) in cells K18:K27 using the formulas above.

  4. Compute the gradient for each input \(\Delta b\) in cells L18-L27. Use absolute and relative cell referencing.

  5. Compute the mean gradient \(\Delta b\) in cell L28.

  6. Compute the gradient for each input \(\Delta w\) in cells M18-M27. Use absolute and relative cell referencing.

  7. Compute the mean gradient \(\Delta w\) in cell M28.

Tuning via Backpropagation#

In “Step 1” of the worksheet, we will use backpropagation to train the neural network using our data.

  1. In cell A33, compute the new Bias by adding the old Bias (cell A18) to the gradient \(\Delta b\) (cell L28).

  2. In cell B33, compute the new Weight by adding the old Weight (cell B18) to the gradient \(\Delta w\) (cell M28).

The remaining training steps (2-20) in the worksheet are already set up to update automatically. To manually add more training steps, copy and paste the existing rows and verify that all cell references update correctly.

Back Propagation

For more information on backpropagation, review the PCMs: Neural Networks

Task 6: Machine Learning using a General Linear Model#

To predict future data, we can use several different types of machine learning. Rather than using a neural network, you can also use a General Linear Model (GLM) to directly calculate the slope and y-intercept using the Least-Squares Method. To determine the equation of the line, you must find the slope and intercept values that minimize the prediction error, which is also known as the cost function or sum of squared errors (SSE).

Because there is more than one unknown (slope and y-intercept), partial derivatives will need to be used. This is done below.

(7.10)#\[c(a,b) = \text{SSE} = \sum^n_{i=1}[y_i-f(x_1)]^2\]
(7.11)#\[\frac{\delta SSE}{\delta a} = -2 \sum^n_{i=1}(y_i - a - bx_i) \, \text{and} \, \frac{\delta SSE}{\delta b} = - 2 \sum^n_{i=1}(y_i - a - bx_i)\]

Through algebraic manipulation, the following two equations are used to directly calculate the line of best fit with the lowest SSE.

(7.12)#\[b = \frac{N \sum^n_{i=1} x_iy_i - \sum^n_{i=1}x_i \cdot \sum^n_{i=1} y_i}{N \sum^n_{i=1} x^2_i - (\sum^n_{i=1}x_i)^2}\]
(7.13)#\[a = \bar{y} -b \bar{x}\]

These equations are already programmed into Excel as built-in functions =SLOPE() and =INTERCEPT().

Linear Regression Calculations

For more information on linear regression, least-squares method, and SSE calculations, and Excel built-in functions review the PCMs: Linear Regression

Instructions#

Navigate to the “General Linear Model” tab in your spreadsheet.

  1. Use Excel’s built-in functions =SLOPE() and =INTERCEPT() in cells C16 and C17 to calculate the parameters.

  2. In column D, use absolute and relative cell referencing to calculate the predicted values \(f(x_i)\) for the resistance of the copper wire with cross-sectional area of \(6 \ mm^2\) at different lengths.

  3. Calculate the square error in column E using the formula \(e^2_i = (f(x_i)-y_i)^2\).

  4. Calculate the sum of squared errors (SSE) in cell H3 by summing the square errors in column E.

    Quick check: This value should be identical to the cost function of the neural network.

  5. Create a scatter plot of the wire length and resistance data. Insert a trend line and display the equation on the figure. Make sure to format the figure for technical presentation.

Final Deliverables#

Complete your own analysis and submit your own responses for the report.

  • Design Challenge Individual Report: Complete this document independently, which must include:

    • Your name and section number.

    • Task 1: Your pseudocode flowchart image.

    • Tasks 2 and 3: Your reflection on the decision tree algorithm optimization.

    • Task 4: Your physical system block diagram image.

    • Tasks 5 and 6: Your responses to all machine learning interpretation questions, as well as your formatted GLM scatter plot.

  • Decision Tree Excel File: Your completed 7.3.1_Decision_Tree_Template.xlsx file.

  • Neural Network Excel File: Your completed 7.3.2_Neural_Network_Template.xlsx file.