Perspective Chapter: Linear Regression and Logistic Regression Models

Abstract

In this chapter, we have discussed the detailed concept of simple linear regression and logistic regression analysis. Further we have discussed the procedure of computing regression coefficients, standard error, t test, Z test, p value and 95% confidence intervals for simple linear regression and logistic regression analysis. We also explained that for testing the simple linear regression coefficient, we use t test, whereas, for testing the logistic regression coefficient, we use Z test. Several examples on medical data are considered and various related statistics were computed using manually, R studio package, and Jamovi.

1. Introduction

The method of linear regression is used in predicting the value of one variable based on the value of another variable. The variable you need to predict is known as dependent variable, whereas the variable used to predict the other variable’s value is known as independent variable. This method contains one or more than one explanatory variables. If it contains only one explanatory variable is called simple linear regression, otherwise, multiple linear regression model. The regression coefficients involved in linear equation is estimated using the least squares method of estimation. Once regression coefficients are estimated, you can fit a model that predicts the value of dependent variable. Linear regression is used to establish a linear relation between response and explanatory variable in biological, behavioral, environmental, social sciences, business etc. Montogomery et al. [1], Rencher and Bruce [2], Swaminathan [3] and Lane [4] discussed regression analysis with several examples. Further, Noce and McKeown [5] discussed a logistic modeling of factors influencing internet use. While Seo et al. [6] discussed the relations between physical activity and behavioral.

Logistic regression is a statistical method that is used to establish a relationship between one dependent variable and one or more than one explanatory variables, where dependent variable is dichotomous.

2. Simple linear regression

Suppose random samples ($x_{i1},x_{i2},\dots ,x_{\mathit{\text{in}}},y_1)$ of size n, where i = 1, 2, …, n; is drawn from a population. The random variables ($x_1,x_2,\dots ,x_n)$ are generally known as predictor variables. However, depending upon situation and with practical point of view, these random variables are also known with different names. The different names are independent variables, covariates, regressor and explanatory variables. Variable y is called as response variable. Sometimes variable y is also known as dependent variable or outcome variable. Suppose we are willing to establish a linear regression between response variable y and explanatory variables ($x_1,x_2,\dots ,x_n),$then it could be represented by a model,

$$\mathrm{y}={\beta }_0+{\beta }_1{x}_1+{\beta }_2{x}_2+\dots +{\beta }_n{x}_n+\mathrm{e}$$E1

where, ${\beta }_0$, ${\beta }_1,{\beta }_2,\dots {\beta }_n$ are parameters and is known as regression coefficients, e is random error which is distributed normally with mean zero and variance ${\sigma }^2.$

For example, the effect of age, weight, height and walking habit on systolic blood pressure. This model is called multiple regression models as predicted variables are more than one.

Suppose we are interested in bi-variate regression model, where Y is response variable and X is predicted variable. This model is called simple linear regression model or general linear model. This model is represented by

$$\mathrm{Y}={\beta }_0+{\beta }_{1}X+e$$E2

Where, Y is response variable, X is predicted variable, ${\beta }_0$ is intercept, ${\beta }_1$ is regression coefficient and e is random variable; and e is distributed normally with mean zero and variance ${\sigma }^2.$ For example, the effect of age on systolic blood pressure.

3. Scatter diagram

A scatter diagram is a two-dimension graph involving the magnitude of the response variable (Y) and predicted variable (X). Scatter diagram provides a rough idea about the relationship between response and predicted variables. Following are the various steps for drawing a scatter diagram:

1. Select the horizontal axis (X) and vertical axis (Y)

2. Take response variable on Y-axis and predicted variable on X-axis.

3. Tick the point at the corresponding area of (X, Y).

In the scatter diagram, if the observations are approximately scattered around the straight line, it shows a linear relationship between response and predicted variables. Once the relationship is established, one can use simple linear regression model to know the relationship between the variables. However, if the observations are not scattered around the straight line, it does not show a linear relationship between the two variables. In such situation, one can use transformation or non-linear regression method to find the best fitted regression model.

Example 1: A sample of 15 men of age group 30–70 was collected to investigate the effect of weight of the patients on the sugar level of the diabetic patients. The data on the blood sugar level (mg/dl) and weight (in kg) of 15 men are shown in Table 1.

S. no.	Blood sugar level	Weight of the patients (in kg)
1	146	50
2	145	48
3	141	46
4	168	69
5	132	40
6	190	80
7	180	70
8	130	38
9	181	75
10	148	59
11	110	30
12	147	54
13	146	51
14	120	35
15	155	65

Table 1.

Blood sugar level and duration of walk of 15 men.

Draw the scatter diagram and give your interpretation about the data.

From Figure 1, it is clear that approximately all the 15 observations are scatted around the straight line. Hence, there is linear relationship between blood sugar level and weight of the person.

Figure 1.

Scatter diagram.

3.1 Assumptions underlying linear regression model

For applying any statistical method, first of all, we should study the assumptions underlying it. So we shall discuss the assumptions of simple linear regression models. Following are the assumptions:

1. The regression model is assumed to be linear in parameters.

2. The error term e is assumed to be normally distributed with mean 0 and variance ${\mathrm{\sigma }}^2$, i.e., e σ ∼ N(0, ${\mathrm{\sigma }}^2$).

3.2 Estimation of parameters

For regression model (2), ${\beta }_1$ is the parameter and is constant and unknown which can be estimated using least squares method of estimation. Model (2) can be rewritten as

$${\mathit{y}}_{\mathit{i}}\mathbf{=}{\beta }_0+{\beta }_1{x}_i+e_i,\text{where}\mathrm{i}=1,2,\dots ,\mathrm{n}.$$E3

$$\mathrm{E}=\sum _{i=1}^n\underset{i}{\overset{2}{e}}=\sum _{i=1}^n{\left(y_i-{\beta }_0-{\beta }_1{x}_i\right)}^2$$E4

On differentiating the error sum of squares (E) with respect to ${\beta }_0$ and ${\beta }_1$ and then equating them to zero, we can obtain the least squares estimator of ${\beta }_0$ and ${\beta }_1$.

$$\frac{\mathit{\delta E}}{{\mathit{\delta \beta }}_0}=-2\sum _{i=1}^n\left(y_i-{\beta }_0-{\beta }_1{x}_i\right)=0$$E5

$$\sum _{i=1}^n\left(y_i-{\beta }_0-{\beta }_1{x}_i\right)=0$$E6

$$\mathrm{So},\sum _{i=1}^n{y}_i=n{\beta }_0+{\beta }_1\sum _{i=1}^n{x}_i$$E7

$$\text{Similarly},\frac{\mathit{\delta E}}{{\mathit{\delta \beta }}_1}=-2\sum _{i=1}^n\left(y_i-{\beta }_0-{\beta }_1{x}_i\right)x_i=0$$E8

$$\sum _{i=1}^n\left(y_i{x}_i-{\beta }_0{x}_i-{\beta }_1\underset{i}{\overset{2}{x}}\right)=0$$E9

$$\mathrm{So},\sum _{i=1}^n{y}_i{x}_i={\beta }_0\sum _{i=1}^n{x}_i+{\beta }_1\sum _{i=1}^n\underset{i}{\overset{2}{x}}$$E10

On solving (7) and (10), we get

$$\widehat{{\beta }_1}\left[\mathrm{n}\sum _{i=1}^n\underset{i}{\overset{2}{x}}-{\left(\sum _{i=1}^n{x}_i\right)}^2\right]=\mathrm{n}\sum _{i=1}^n{y}_i{x}_i-\sum _{i=1}^n{y}_i\sum _{i=1}^n{x}_i$$E11

$$\text{Hence},\widehat{{\beta }_1}=\frac{\mathrm{n}{\sum }_{i=1}^n{y}_i{x}_i-{\sum }_{i=1}^n{y}_i{\sum }_{i=1}^n{x}_i}{\mathrm{n}{\sum }_{i=1}^n{x}_i^2-{\left({\sum }_{i=1}^n{x}_i\right)}^2}$$E12

$$\text{Or},\widehat{{\beta }_1}=\frac{{\sum }_{i=1}^n{y}_i{x}_i-n\overline{y}\overline{x}}{{\sum }_{i=1}^n{x}_i^2-n{\overline{x}}^2}$$E13

Eq. (12) can easily be written as.

$$\widehat{{\beta }_1}=\frac{{\sum }_{i=1}^n\left(x_i-\overline{x}\right)\left(y_i-\overline{y}\right)}{{\sum }_{i=1}^n{\left(x_i-\overline{x}\right)}^2}$$E14

On dividing the numerator and denominator of the above equation by n, we have,

$$\widehat{{\beta }_1}=\frac{\mathit{Cov}\left(X,Y\right)}{\mathit{Var}\left(X\right)}$$E15

From (7), we have n$\overline{y}$ = n$\widehat{{\beta }_0}$ + n $\widehat{{\beta }_1}\overline{x}$

$$\mathrm{So},\widehat{{\beta }_0}=\overline{y}-\widehat{{\beta }_1}\overline{x}$$E16

Here, $\widehat{{\beta }_0}$ and $\widehat{{\beta }_1}$ are the least square estimator of intercept ${\beta }_0$ and slope ${\beta }_1$. In the regression model, slope is called regression coefficient. Hence, on ward ${\beta }_1$ will be called regression coefficient. Thus, for the linear regression method, the fitted regression model is given by

$$\widehat{y_i}=\widehat{{\beta }_0}+\widehat{{\beta }_1}{x}_i$$E17

In the matrix notation, the linear model can be written as

$$\mathrm{Y}=\mathrm{X\beta }+\mathrm{e}$$E18

Where, Y is a vector of n × 1 observations, X is a matrix of n × 2, β is a vector of 2 × 1 parameters, and e is the random error of n × 1.

Using least squares method of estimation, the normal equation is obtained as

$$X^{\prime }\mathrm{Y}=X^{\prime }\mathit{X\beta }$$E19

On multiplying both side of (19) by ${\left(X^{\prime }X\right)}^{-1}$, we have

$${\left(X^{\prime }X\right)}^{-1}\left(X^{\prime }\mathrm{Y}\right)={\left(X^{\prime }X\right)}^{-1}\left(X^{\prime }X\right)\beta$$E20

$$\text{Hence},\widehat{\beta }={\left(X^{\prime }X\right)}^{-1}\left(X^{\prime }\mathrm{Y}\right)$$E21

Where, $\widehat{\beta }$ is the estimate of the regression coefficient $\beta .$

In that case, the fitted regression model is given by

$$\widehat{Y}=\widehat{{\beta }_0}+\widehat{{\beta }_1}X$$E22

Example 2: A samples of 15 men of age group 30–70 was collected to investigate the effect of weight (in kg) of the patients on the blood pressure level of the diabetic patients. The data on the blood pressure level (mm/hg) and weight (in kg) of 15 men are shown in Tables 2 and 3.

S. no.	Blood pressure level (mm/hg)	Weight (in kg)
1	125	50
2	123	48
3	120	46
4	181	73
5	105	40
6	190	80
7	185	75
8	118	45
9	175	74
10	168	69
11	110	43
12	130	54
13	128	51
14	116	44
15	179	72

Table 2.

Blood pressure level and weight of 15 men.

y	x	y square	x square	xy	Predicted value of y	Residual	Residual sum of squares
125	50	15,625	2500	6250	126.8429	−1.84293	3.396390985
123	48	15,129	2304	5904	122.4507	0.549282	0.301710716
120	46	14,400	2116	5520	118.0585	1.941494	3.769398952
181	73	32,761	5329	13,213	177.3534	3.646632	13.29792494
105	40	11,025	1600	4200	104.8819	0.11813	0.013954697
190	80	36,100	6400	15,200	192.7261	−2.72611	7.431675732
185	75	34,225	5625	13,875	181.7456	3.25442	10.59124954
118	45	13,924	2025	5310	115.8624	2.1376	4.56933376
175	74	30,625	5476	12,950	179.5495	−4.54947	20.69771368
168	69	28,224	4761	11,592	168.5689	−0.56894	0.323697275
110	43	12,100	1849	4730	111.4702	−1.47019	2.161452755
130	54	16,900	2916	7020	135.6274	−5.62735	31.66711304
128	51	16,384	2601	6528	129.039	−1.03904	1.079595809
116	44	13,456	1936	5104	113.6663	2.333706	5.446183694
179	72	32,041	5184	12,888	175.1573	3.842738	14.76663534
2153	864	322,919	52,622	130,284	2153		119.5140309

Table 3.

Predicted and residual value.

Where, residual = (y – predicted value of y).

$$\begin{array}{ll}\text{Regression coefficient},\widehat{{\beta }_1}& =\frac{\mathrm{n}{\sum }_{i=1}^n{y}_i{x}_i-{\sum }_{i=1}^n{y}_i{\sum }_{i=1}^n{x}_i}{\mathrm{n}{\sum }_{i=1}^n{x}_i^2-{\left({\sum }_{i=1}^n{x}_i\right)}^2}\\ & =\frac{15\mathrm{x}130284-2153x864}{15\mathrm{x}52622-{\left(864\right)}^2}=\frac{1954260-1860192}{789330-746496}=\frac{94068}{42834}\end{array}$$E23

Hence, $\widehat{{\beta }_1}=2.196106.$

$$\begin{array}{c}\widehat{{\beta }_0}=\overline{y}-\widehat{{\beta }_1}\overline{x}=143.5333–2.196106\times 57.6\\ =17.03763.\end{array}$$

The fitted regression model is given by

$$\widehat{Y}=17.03763+2.196106X$$E24

r = 0.996 and $R^2$ = 0.991.

From model coefficient (Table 4), it is obvious that regression coefficient of predictor weight is highly significant as p < 0.001. Also $R^2$ is very close to 1. This concludes that as the weight of the patient increases, blood pressure increases. That is, weight is under control, the blood pressure is normal or under normal. In this example, weight ranges from 69 to 80, the blood pressure ranges from 179 to 190. However, weight ranges from 40 to 54, the blood pressure ranges from 105 to 130. Again, when the weight ranges from 69 to 80, the blood pressure ranges from 168 to 190. Thus, higher the weight, higher is blood pressure.

Predictor	Estimate	SE	t	P
Intercept	17.03763	3.3607	5.07	<0.001
β	$2.196106$	0.0567	38.70	<0.001

Table 4.

Model coefficient.

3.3 Regression coefficient using R studio package

>y = c(125,123,120,181,105,190,185,118,175,168,110,130,128,116,179)

>x = c(50,48,46,73,40,80,75,45,74,69,43,54,51,44,72)

>result = data.frame(y,x)

> z = lm(y ∼ x,result)

> summary(z)

4. Forecasting or predicted value of Y

Using the fitted model (24), we can easily forecast the blood pressure level corresponding to its weight. Suppose, weight of a patient is 82 kg then its blood pressure will be obtained from $\widehat{Y}$= 17.03763 + $2.196106\times 82$ = 197.1183.

Residuals: We can obtain the residuals of all the 15 patients using (Y – $\widehat{Y})$. This value is shown in Table 3.

R Squares: We can compute $R^2$ using the residual sum of squares from the expression

$$R^2=1-\frac{\mathit{\text{Residual}}\mathit{sum}\mathit{\text{of squares}}}{\mathit{\text{total}}\mathit{sum}\mathit{\text{of squares of}}y}$$E25

Where, residual sum of squares = $\sum {\left(\mathrm{Y}-\widehat{Y}\right)}^2$ = 119.5140309 and is shown in Table 3.

Total sum of squares of y = ${\sum }_{i=1}^n{y}_i^2-\frac{{\left({\sum }_{i=1}^n{y}_i\right)}^2}{n}$ = 322,919 – $\frac{{\left(2153\right)}^2}{15}$ = 13891.73333.

$\mathrm{So},R^2=1-\frac{119.5140309}{13891.73333}=0.9914.$

We can also obtain Regression coefficients using the Matrix form as following:

$$X^{\prime }=\left[\begin{array}{ccccccccccccccc}1& 1& 1& 1& 1& 1& 1& 1& 1& 1& 1& 1& 1& 1& 1\\ 50& 48& 46& 73& 40& 80& 75& 45& 74& 69& 43& 54& 51& 44& 72\end{array}\right]$$

$$\begin{array}{l}{X}^{\prime }X=\left[\begin{array}{cc}15& 864\\ 864& 52622\end{array}\right],X^{\prime }Y=\left[\begin{array}{c}2153\\ 130284\end{array}\right]\text{and}\\ {\left(X^{\prime }X\right)}^{-1}=\left[\begin{array}{cc}1.22851006& -0.0201708923\\ -0.0201708923& 0.0003501891\end{array}\right]\end{array}$$

$$\widehat{\beta }={\left(X^{\prime }X\right)}^{-1}{X}^{\prime }Y=\left[\begin{array}{c}17.037634\\ 2.196106\end{array}\right].\widehat{{\beta }_0}=17.037634\text{and}\widehat{{\beta }_1}=2.196106.$$

4.1 R studio program to obtain the estimate of regression coefficients

>X = matrix(c(1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,50,48,46,73,40,80,75,45,74,69,43,54,51,44,72),ncol = 2)

$>X^{\prime }$= t(X)

$>X^{\prime }X$ = $X^{\prime }\%\ast \%X$

>Y = c(125,123,120,181,105,190,185,118,175,168,110,130,128,116,179)

>m = solve($X^{\prime }X)$

$>X^{\prime }Y$= $X^{\prime }$%*%Y

>n=$X^{\prime }Y$

>β=$m\%\ast \%n$

5. Regression lines

Let the regression model is denoted by Y = a + bX, then the two types of regression lines are following:

1. Regression line of Y on X and is denoted by $b_{\mathit{YX}}$

2. Regression line of X on Y and is denoted by $b_{\mathit{XY}}$

$$b_{\mathit{YX}}=\frac{\mathrm{n}{\sum }_{i=1}^n{y}_i{x}_i-{\sum }_{i=1}^n{y}_i{\sum }_{i=1}^n{x}_i}{\mathrm{n}{\sum }_{i=1}^n{x}_i^2-{\left({\sum }_{i=1}^n{x}_i\right)}^2}=2.196106$$E26

$$\text{Regression line of X on Y and is denoted by}{b}_{\mathit{XY}}=\frac{\mathrm{n}{\sum }_{i=1}^n{y}_i{x}_i-{\sum }_{i=1}^n{y}_i{\sum }_{i=1}^n{x}_i}{\mathrm{n}{\sum }_{i=1}^n{y}_i^2-{\left({\sum }_{i=1}^n{y}_i\right)}^2}=0.4514$$E27

6. Expectation and variance of estimators

$$\mathrm{E}\left(\widehat{{\beta }_0}\right)={\beta }_0\text{and}\mathrm{E}\left(\widehat{{\beta }_1}\right)={\beta }_1$$E28

This show that least squares estimators $\widehat{{\beta }_0}$ and $\widehat{{\beta }_1}$ are unbiased estimators of ${\beta }_0$ and ${\beta }_1$, respectively.

$$\mathrm{V}\left(\widehat{{\beta }_0}\right)={\sigma }^2\left[\frac{1}{n}+\frac{{\overline{x}}^2}{\mathit{sum}\mathit{\text{of squares of}}x}\right]$$E29

$$\text{And}\mathrm{V}\left(\widehat{{\beta }_1}\right)={\left(X^{\text{'}}X\right)}^{-1}{\sigma }^2=\frac{{\sigma }^2}{\mathit{sum}\mathit{\text{of squares of}}x}.$$E30

Where, ${\sigma }^2$ is unknown. It is estimated from the given data as

$\widehat{{\sigma }^2}$ = $\left(\frac{y_i-{\widehat{y_i}}^2}{n-2}\right)$, and sum of squares of x is obtained as

Sum of squares of x = ${\sum }_{i=1}^n{x}_i^2-\frac{{\left({\sum }_{i=1}^n{x}_i\right)}^2}{n}$; where n is number of observations.

Thus, when ${\sigma }^2$ is unknown, V($\widehat{{\beta }_0})$ and V($\widehat{{\beta }_1})$ is determined as

$$\mathrm{V}\left(\widehat{{\beta }_0}\right)=\widehat{{\sigma }^2}\left[\frac{1}{n}+\frac{{\overline{x}}^2}{\mathit{sum}\mathit{\text{of squares of}}x}\right]\text{and}$$E31

$$\mathrm{V}\left(\widehat{{\beta }_1}\right)=\frac{\widehat{{\sigma }^2}}{\mathit{sum}\mathit{\text{of squares of}}x}.$$E32

Again, when ${\sigma }^2$ is unknown, standard error of $\widehat{{\beta }_0}$ and $\widehat{{\beta }_1}$ are determined as

$$\mathrm{SE}\left(\widehat{{\beta }_0}\right)=\sqrt{\mathrm{V}\left(\widehat{{\beta }_0}\right)}=\sqrt{\widehat{{\sigma }^2}\left[\frac{1}{n}+\frac{{\overline{x}}^2}{\mathit{sum}\mathit{\text{of squares of}}x}\right]}\text{and}$$E33

$$\mathrm{SE}\left(\widehat{{\beta }_1}\right)=\sqrt{\mathrm{V}\left(\widehat{{\beta }_1}\right)}=\sqrt{\frac{\widehat{{\sigma }^2}}{\mathit{sum}\mathit{\text{of squares of}}x}}.$$E34

Example 3: Consider the example 2, compute the estimate of V($\widehat{{\beta }_0}$), V($\widehat{{\beta }_1}$) and its standard error.

Solution: From example 2, we have $\widehat{\widehat{{\beta }_0}=17.037634\mathit{\text{and}}}\widehat{{\beta }_1}$ = 2.196106. Sum of squares of X = ${\sum }_{i=1}^n{x}_i^2-\frac{{\left({\sum }_{i=1}^n{x}_i\right)}^2}{n}$ = 52,622 − $\frac{{864}^2}{15}$ = 52,622 – 49766.4 = 2855.6.

$$\widehat{{\sigma }^2}=\frac{{\sum }_{i=1}^n(y_i-{\widehat{y_i}}^2)}{n-2}=\frac{119.5140}{13}=9.1934.$$

$$\overline{x}=\frac{{\sum }_{i=1}^n{x}_i}{n}=\frac{864}{15}=57.6.$$

Estimates of the V($\widehat{{\beta }_0})$ and V($\widehat{{\beta }_1})$ can be determined from

$$\begin{array}{c}\mathrm{V}\left(\widehat{{\beta }_0}\right)=\widehat{{\sigma }^2}\left[\frac{1}{n}+\frac{{\overline{x}}^2}{\mathit{sum}\mathit{\text{of squares of}}x}\right]\\ =9.1934\left[\frac{1}{15}+\frac{{\left(57.6\right)}^2}{2855.6}\right]\end{array}$$E35

$$\mathrm{So},\mathrm{V}\left(\widehat{{\beta }_0}\right)=11.2941844.$$

$$\mathrm{V}\left(\widehat{{\beta }_0}\right)=\frac{\widehat{{\sigma }^2}}{\mathit{sum}\mathit{\text{of squares of}}x}==\frac{9.1934}{2855.6}=0.003219428$$E36

$$\mathrm{SE}\left(\widehat{{\beta }_0}\right)=\sqrt{\mathrm{V}\left(\widehat{{\beta }_0}\right)}=\sqrt{11.2941844}=3.360682133,$$

$$\mathrm{SE}\left(\widehat{{\beta }_1}\right)=\sqrt{\mathrm{V}\left(\widehat{{\beta }_1}\right)}=\sqrt{0.003219428}=0.056740004.$$

7. Testing of hypothesis of estimated regression coefficients $\widehat{{\beta }_0}$ and $\widehat{{\beta }_1}$

For testing the hypothesis that the sample comes from the population for which the value of ${\beta }_0$ is equal to 0. That is,

Our Null hypothesis $H_0:{\beta }_0=0$

Against the alternate hypothesis $H_1:{\beta }_0\ne 0$.

Under the null hypothesis, for testing $H_0:{\beta }_0=0$, we define the test statistics given by

$$\begin{array}{c}\mathrm{t}=\frac{\widehat{{\beta }_0}-E\left(\widehat{{\beta }_0}\right)}{\mathit{SE}\left(\widehat{{\beta }_0}\right)}=\frac{\widehat{{\beta }_0}-{\beta }_0}{\mathit{SE}\left(\widehat{{\beta }_0}\right)}=\frac{\widehat{{\beta }_0}-0}{\mathit{SE}\left(\widehat{{\beta }_0}\right)}\end{array}$$E37

$$\begin{array}{c}=\frac{\widehat{{\beta }_0}}{\sqrt{\widehat{{\sigma }^2}\left[\frac{1}{n}+\frac{{\overline{x}}^2}{\mathit{sum}\mathit{\text{of squares of}}x}\right]}},\text{where}{\sigma }^2\text{is unknown}.\end{array}$$E38

Similarly, for testing the hypothesis that the sample comes from the population for which the value of ${\beta }_1$ is equal to 0. That is,

Our Null hypothesis $H_0:{\beta }_1=0$.

Against the alternate hypothesis $H_1:{\beta }_1\ne 0$.

Thus, Under the null hypothesis, for testing $H_0:{\beta }_1=0$, we define the test statistics given by

$$\mathrm{t}=\frac{\widehat{{\beta }_1}-E\left(\widehat{{\beta }_1}\right)}{\mathit{SE}\left(\widehat{{\beta }_1}\right)}=\frac{\widehat{{\beta }_1}-{\beta }_1}{\mathit{SE}\left(\widehat{{\beta }_1}\right)}=\frac{\widehat{{\beta }_1}-0}{\mathit{SE}\left(\widehat{{\beta }_1}\right)}=\frac{\widehat{{\beta }_1}}{\sqrt{\frac{\widehat{{\sigma }^2}}{\mathit{sum}\mathit{\text{of squares of}}x}}}$$E39

where ${\sigma }^2$ is unknown, and statistics t follows student’s t distribution with (n – 2) degrees of freedom.

Example 4: Consider the data given in example 2. Find the effect of weight on blood pressure of 15 patients, test the null hypothesis for testing the significance of the regression coefficients $\widehat{{\beta }_0}$ and $\widehat{{\beta }_1}$ at 5% level of significance.

Solution: From example 2, we have,

$$\widehat{{\beta }_0}=17.037634,\mathrm{SE}\left(\widehat{{\beta }_0}\right)=3.360682133,\text{and}$$

$$\widehat{{\beta }_1}=2.196106,\mathrm{SE}\left(\widehat{{\beta }_1}\right)=0.056740004.$$

Under the null hypothesis, for testing $H_0:{\beta }_0=0$, we have,

$$\mathrm{t}=\frac{\widehat{{\beta }_0}-E\left(\widehat{{\beta }_0}\right)}{\mathit{SE}\left(\widehat{{\beta }_0}\right)}=\frac{\widehat{{\beta }_0}-{\beta }_0}{\mathit{SE}\left(\widehat{{\beta }_0}\right)}=\frac{\widehat{{\beta }_0}-0}{\mathit{SE}\left(\widehat{{\beta }_0}\right)}$$E40

where ${\sigma }^2$ is unknown.

$$\mathrm{t}=\frac{17.037634}{3.360682133}=5.069695.$$

Under the null hypothesis, for testing $H_0:{\beta }_1=0$, we have

$$\mathrm{t}=\frac{\widehat{{\beta }_1}-E\left(\widehat{{\beta }_1}\right)}{\mathit{SE}\left(\widehat{{\beta }_1}\right)}=\frac{\widehat{{\beta }_1}-{\beta }_1}{\mathit{SE}\left(\widehat{{\beta }_1}\right)}=\frac{\widehat{{\beta }_1}-0}{\mathit{SE}\left(\widehat{{\beta }_1}\right)},\text{where}{\sigma }^2\text{is unknown}.$$E41

$$\mathrm{t}=\frac{2.196106}{0.056740004}=38.70472.$$

With α = 0.05, for two sided α, the tabulated value of t at 13 degrees of freedom with 5% level of significance is 2.160. In case of $H_0:{\beta }_0=0$, the calculated value of t (=5.069695) is greater than tabulated value of t, so test is significant. That is, we reject the null hypothesis. Hence, we may conclude that the value of ${\beta }_0$ is not equal to zero.

Similarly, In case of $H_0:{\beta }_1=0$, the calculated value of t (=38.70472) is greater than tabulated value of t, so test is highly significant. That is, we reject the null hypothesis. Hence, we may conclude that the value of ${\beta }_1$ is not equal to zero. Thus, the fitted simple regression model is highly significant. In other word, we can say as the weight of the patient increases, the chance of blood pressure may increase.

Alternatively, we can also test the significance of regression coefficient ${\beta }_1$ using the analysis of variance Table 5.

Sources of variation	Degrees of freedom	Sum of squares	Mean squares	F-Ratio	P-value
Regression	1	$\sum {\left(\widehat{y_i}-\overline{y}\right)}^2$	SSR/df = MSR	MSR/MSE
Error	(n − 2)	$\sum {\left(y_i-\widehat{y_i}\right)}^2$	SSE/df = MSE
Total	(n – 1)	$\sum {\left(y_i-\overline{y}\right)}^2$

Table 5.

Analysis of variance.

Example 5: Consider the data given in example 2. Find the effect of weight on blood pressure of 15 patients, test the null hypothesis for testing the significance of the regression coefficients $\widehat{{\beta }_1}$ at 5% level of significance using analysis of variance Table 5.

Using Table 3, we have Table 6.

y	x	Predicted value of y ($\widehat{y_i})$	$\sum {\left(\widehat{y_i}-\overline{y}\right)}^2$	$\sum {\left(y_i-\widehat{y_i}\right)}^2$	$\sum {\left(y_i-\overline{y}\right)}^2$
125	50	126.8429	278.568451	3.396390985	343.4832089
123	48	122.4507	444.475264	0.301710716	421.6164089
120	46	118.0585	648.965129	3.769398952	553.8162089
181	73	177.3534	1143.797	13.29792494	1403.753609
105	40	104.8819	1493.93304	0.013954697	1484.815209
190	80	192.7261	2419.93256	7.431675732	2159.154209
185	75	181.7456	1460.17834	10.59124954	1719.487209
118	45	115.8624	765.678707	4.56933376	651.9494089
175	74	179.5495	1297.16479	20.69771368	990.1532089
168	69	168.5689	626.78347	0.323697275	598.6194089
110	43	111.4702	1028.04315	2.161452755	1124.482209
130	54	135.6274	62.5039822	31.66711304	183.1502089
128	51	129.039	210.083689	1.079595809	241.2834089
116	44	113.6663	892.038047	5.446183694	758.0826089
179	72	175.1573	1000.07497	14.76663534	1257.886809
2153	864	2153	13772.2206	119.5140309	13891.73333

Table 6.

Regression, error and total sum of squares.

Using Table 6, we can obtain the ANOVA table as shown in Table 7.

Sources of variation	Degrees of freedom	Sum of squares	Mean squares	F-Ratio	P-value
Regression	1	13772.2206	13772.2206	1498.057	<0.001
Error	13	119.5140309	9.193387
Total	14	13891.73333

Table 7.

Analysis of variance.

From Table 7, we can observe the p value for regression coefficient is less than 0.001 as well calculated value of F is very large. Which is greater than tabulated value of F with (1, 13) degrees of freedom at 5% level of significance, where tabulated value of F is 4.67. This shows that the test is significant and hence rejects the null hypothesis.

7.1 R studio program for obtaining ANOVA table

>y = c(125,123,120,181,105,190,185,118,175,168,110,130,128,116,179)

>x = c(50,48,46,73,40,80,75,45,74,69,43,54,51,44,72)

>result = data.frame(y,x)

> av. = aov(y ∼ x,result)

> summary(av)

8. Confidence interval of estimated regression coefficients

Now we discuss how to obtain 100% confidence interval of estimated regression coefficients. In fact we are interested for confidence interval for regression coefficient ${\beta }_1$ only. In fact we generally compute the confidence interval to determine the range. In case of regression coefficient ${\beta }_1$, we wish to determine the lower and upper limit of the ${\beta }_1$. We can obtain the 100% confidence interval of the ${\beta }_1$as

Confidence interval for ${\beta }_1$= $\widehat{{\beta }_1}$ ± $t_{\left(n-2\right),\alpha }$ SE($\widehat{{\beta }_1}$) for two tailed test.

Example 6: Consider the data given in example 2. Find the effect of weight on blood pressure of 15 patients. Obtain 100% confidence interval of regression coefficients $\widehat{{\beta }_1}$.

Solution: For this data from example 2, we have,

$$\widehat{{\beta }_1}=2.196106,\mathrm{SE}\left(\widehat{{\beta }_1}\right)=0.056740004\text{and}{t}_{\left(n-2\right),\alpha }=2.160.$$

$$\begin{array}{ll}\text{Confidence interval for}{\beta }_1& =2.196106\pm 2.160\times 0.056740004\\ & =2.196106\pm 0.122558409\end{array}$$

So, lower confidence limit of ${\beta }_1$ is 2.196106 − 0.122558409 = 2.073547489.

Upper confidence limit of ${\beta }_1$ is 2.196106 + 0.122558409 = 2.318664306.

Thus, the confidence limit of ${\beta }_1$ is ranges from 2.073547489 to 2.318664306.

If one is interested for determining the confidence limit of ${\beta }_0$, the same procedure can be used.

9. Logistic regression

Logistic regression is used to obtain odds ratio in the presence of more than one explanatory variable. The procedure is quite similar to multiple linear regressions with the exception that the response variable is binomial. The result is the impact of each variable on the odds ratio of the observed event of interest

We can also say that Logistic regression is used for predicting binary outcomes on the basis of one or more predictor variables. The concept of logistic regression is similar to the ordinary multiple linear regression. In this method we are willing to fit a best model which can determine the relationship between a response variable and one or more explanatory variables. As in the case of ordinary linear regression, the form of the model is linear with respect to the regression parameters, the same is true for the logistic regression. The only difference between the two regression is following: in logistic regression the response variable is binary (also called dichotomous), whereas in ordinary linear regression it is continuous. Logistic regression can also be called as a predictive algorithm in which by using explanatory variables one can predict the dependent variable, just like Linear regression, but with a simple difference that the dependent variable in the logistic regression should be considered as a categorical variable.

10. Logistic function

For understanding the logistic regression, first we must determine logistics function.

Let us consider the equation of the best fit model in the simple linear regression as

$$\mathrm{y}={\beta }_0+{\beta }_1\mathrm{x}$$E42

where, y is response variable and x is explanatory variable.

Let us replace y by probability P which is given as

$$\mathrm{P}={\beta }_0+{\beta }_1\mathrm{x}$$E43

In (43) value of P may be negative in some case and in some other cases, value of P may be more than one. However, value of P ranges from 0 to 1 only. This is a contradiction. To overcome this problem, we can take odds of P instead of probability. Odds of probability is defined as odd = $\frac{P}{1-P}$. That is, odds of probability is defined as the ratio of the probability of success and the probability of failure.

So, from (43), we have

$$\frac{P}{1-P}={\beta }_0+{\beta }_1\mathrm{x}.$$E44

As we are aware that odds will be always positive, that is odds are ranging from 0 to infinity. Again, to overcome this problem we take the log transformation because by considering log transformation it will range from − infinity to + infinity.

$$\mathrm{Log}\left(\frac{P}{1-P}\right)={\beta }_0+{\beta }_1\mathrm{x}$$E45

On taking exponential on both sides of (45), we have

$$\mathrm{Exp}\left[\mathrm{Log}\left(\frac{P}{1-P}\right)\right]=exp\left[{\beta }_0+{\beta }_1\mathrm{x}\right]$$E46

That is, $\left(\frac{P}{1-P}\right)$log(e) = $e^{{\beta }_0+{\beta }_1\mathrm{x}}$

$$\left(\frac{P}{1-P}\right)=e^{{\beta }_0+{\beta }_1\mathrm{x}}$$E47

$$\mathrm{P}=\left(1–\mathrm{p}\right)e^{{\beta }_0+{\beta }_1\mathrm{x}}$$

$$\mathrm{P}+\mathrm{P}{e}^{{\beta }_0+{\beta }_1\mathrm{x}}=e^{{\beta }_0+{\beta }_1\mathrm{x}}$$

$$\mathrm{P}\left(1+e^{{\beta }_0+{\beta }_1\mathrm{x}}\right)=e^{{\beta }_0+{\beta }_1\mathrm{x}}$$

$$\mathrm{So},\mathrm{P}=\frac{e^{{\beta }_0+{\beta }_1\mathrm{x}}}{\left(1+e^{{\beta }_0+{\beta }_1\mathrm{x}}\right)}$$E48

On dividing the numerator and denominator of (48) by $e^{{\beta }_0+{\beta }_1\mathrm{x}}$, we have

$$\mathrm{P}=\frac{\left(e^{{\beta }_0+{\beta }_1\mathrm{x}}\right)/\left(e^{{\beta }_0+{\beta }_1\mathrm{x}}\right)}{\left(1+e^{{\beta }_0+{\beta }_1\mathrm{x}}\right)/\left(e^{{\beta }_0+{\beta }_1\mathrm{x}}\right)}=\frac{1}{\left(1+e^{{\beta }_0+{\beta }_1\mathrm{x}}\right)/e^{{\beta }_0+{\beta }_1\mathrm{x}}}$$E49

$$\text{Thus},\mathrm{P}=\frac{1}{\left[1+e^{-\left({\beta }_0+{\beta }_1\mathrm{x}\right)}\right]}$$E50

We can call (50) as a logistic function.

If we consider only one explanatory variable then the graph of simple linear regression will give a straight line however, the graph of logistic regression will be of S shape. This is shown in Figure 2.

Figure 2.

Shape of linear and logistic regression.

Logistic regression can also be called as a predictive algorithm in which by using explanatory variables one can predict the dependent variable, just like Linear Regression, but with a simple difference that the dependent variable in the logistic regression should be considered as a categorical variable.

Example 7: A random sample of 300 women were selected, where 300 women are either suffering with cancer or not. The response of yes will be asked from the 300 women. It is found that 225 women responded yes. The response of yes out of n sample number follows binomial distribution with parameters n and p. Obtain the odds ratio.

Solution: Out of 300 women 225 have responded yes for cancer. So, the sample proportion is

$$\widehat{P}=\frac{225}{300}=0.75.$$

Sample proportion cannot be used for finding logistic regression and hence, we need the odds. Where odds is the ratio of proportion for two outcomes. One outcomes is “yes” and the other outcomes is “no”. Proportion of yes is 0.75, hence, proportion of no is 1 − $\widehat{P}$ = 1 − 0.75 = 0.25.

$$\mathrm{Odd}\text{ratio of}\mathrm{yes}\text{and}\mathrm{no}\text{of women cancer}=\frac{\widehat{P}}{1-\widehat{P}}=\frac{0.75}{0.25}=3.$$

Hence, odds are 3 to 1 that woman has cancer yes to no. Similarly, we can also say odds are 1 to 3, that is, women has cancer no to yes is 1 to 3.

Example 8: The sample proportion of women who were detected as cancer patient is 65%, whereas the sample proportion of men detected as cancer patient is 45%.

In this sample of young adult, it can be observe that the sample proportion of women detected as cancer patient is 20% higher than the sample proportion of men detected as cancer. Now we wish to analyze this data using logistic regression. In this example the predictive variable is sex which is a categorical variable. So we need to use a numeric code. The better way is to use a indicator saying whether the adult is women or not. The indicator function is defined as

$$\mathrm{x}=\left\{\begin{array}{l}1\text{if the person is women}\\ 0,\text{if the person is}\mathrm{men}\end{array}\right.$$E51

Since, the response is given in proportion, so we transform it into odds. There will be two odds, one for women and other for men.

$$\text{Odds for women}\mathrm{are}\text{given}\mathrm{as}\frac{\widehat{P}}{1-\widehat{P}}=\frac{0.65}{1-0.65}=\frac{0.65}{0.35}=1.8571.$$

$$\text{Similarly},\text{odds for}\mathrm{men}\text{is given}\mathrm{as}\frac{\widehat{P}}{1-\widehat{P}}=\frac{0.45}{1-0.45}=\frac{0.45}{0.55}=0.8182.$$

Now we can build the logistic regression model by considering log(odds) as the linear function of the explanatory variable. Hence, logistics model is defined as

$$log\left(\frac{\widehat{P}}{1-\widehat{P}}\right)={\beta }_0+{\beta }_1\mathrm{x},$$E52

where x is explanatory variable, p is the binomial proportion and ${\beta }_0$, ${\beta }_1$are the parameters of the logistic regression model.

Here, there are only two values of x and hence write two equations: one for women and other for men.

$$\text{For women},log\left(\frac{\widehat{P}}{1-\widehat{P}}\right)={\beta }_0+{\beta }_1\times 1$$E53

$$\text{And for}\mathrm{men},log\left(\frac{\widehat{P}}{1-\widehat{P}}\right)={\beta }_0+{\beta }_1\times 0$$E54

Because, there is a ${\beta }_1$in the equation of women as x = 1. This is missing in the equation of men as x = 0.

Therefore, the logistic regression model for women and men are following:

$$\mathrm{Log}\left(1.8571\right)={\beta }_0+{\beta }_1$$E55

$$\mathrm{Log}\left(0.8182\right)={\beta }_0$$

$${\beta }_0+{\beta }_1=0.6190$$E56

$${\beta }_0=-0.20065$$E57

On solving (24) and (25), we have

$$-0.20065+{\beta }_1=0.6190$$E58

Hence, ${\beta }_1$ = 0.6190 + 0.20065 = 0.81965.

Now the fitted logistic regression model is given by

$$\mathrm{Log}\left({\mathit{\text{odds}}}_{\mathit{\text{women}}}\right)={\beta }_0+{\beta }_1$$E59

$$\mathrm{So},{\mathit{\text{odds}}}_{\mathit{\text{women}}}=e^{{\beta }_0+{\beta }_1}$$E60

$$\text{Similarly},{\mathit{\text{odds}}}_{\mathit{men}}=e^{{\beta }_0}$$E61

$$\frac{{\mathit{\text{odds}}}_{\mathit{\text{women}}}}{{\mathit{\text{odds}}}_{\mathit{men}}}=\frac{e^{{\beta }_0+{\beta }_1}}{e^{{\beta }_0}}=e^{{\beta }_1}=e^{0.81965}=2.269705.$$

$${\mathit{\text{odds}}}_{\mathit{\text{women}}}=2.269705\times {\mathit{\text{odds}}}_{\mathit{men}}.$$

That is, we can say that odds of women are 2.269705 times odds of men.

Note that, if we have indicator function as

$$\mathrm{x}=\left\{\begin{array}{l}0\text{if the person is women}\\ 1,\text{if the person is}\mathrm{men}\end{array}\right.$$E62

Then the sign of ${\beta }_1$will be negative. That is,

$$\mathrm{Log}\left({\mathit{\text{odds}}}_{\mathit{\text{women}}}\right)={\beta }_0.\mathrm{So},{\mathit{\text{odds}}}_{\mathit{\text{women}}}=e^{{\beta }_0}$$E63

$$\text{Similarly},{\mathit{\text{odds}}}_{\mathit{men}}=e^{{\beta }_0+{\beta }_1}$$E64

$$\frac{{\mathit{\text{odds}}}_{\mathit{\text{women}}}}{{\mathit{\text{odds}}}_{\mathit{men}}}=\frac{e^{{\beta }_0}}{e^{{\beta }_0+{\beta }_1}}=e^{-{\beta }_1}=e^{-0.81965}=0.440586$$

$${\mathit{\text{odds}}}_{\mathit{\text{women}}}=0.440586\times {\mathit{\text{odds}}}_{\mathit{men}}.$$E65

Therefore, we can say odds of women are 0.440586 times odds of men.

Example 9: Hemoglobin contain of 20 patients corresponding to their age was collected at a hospital to know the relationship between hemoglobin and age. The collected observations are shown in Table 8.

Hb(g/Dl)	Age (Year)	Anemic(1 = yes,0 = no)
11.2	15	1
11.3	21	1
11.5	23	1
16.3	25	0
16.5	26	0
10.1	28	1
9.9	30	1
17.1	32	0
17.2	34	0
17.9	36	0
10.1	38	1
11.6	40	1
18.3	43	0
18.6	46	0
18.9	54	0
19.2	56	0
19.6	58	0
19.9	60	0
16.9	62	0
17.2	69	0

Table 8.

Level of hemoglobin and its corresponding age.

If we use a simple linear regression to find the effect of age on the response variable Hemoglobin, we obtain the following statistics using software (Table 9).

Predictor	Estimate	SE	t	P
Intercept	9.382	1.7875	5.25	<0.001
Age	0.153	0.0420	3.64	0.002

Table 9.

Model coefficient and 95% confidence.

Here regression coefficient is significant. That is, there is significant effect of age on hemoglobin.

As we are aware that the amount of hemoglobin in whole blood is expressed in grams per deciliter (g/dl). The normal Hb level for males is 14 to 18 g/dl; and for females is 12 to 16 g/dl. When the hemoglobin level is low, the patient has anemia.

If we are interested to know whether the patient is suffering with anemia then we have to use the logistic regression method. For this we have to transform the Hemoglobin data into presence or absence of Anemia. Since, the data belongs to women patients, so if the value is less than 12, the code is 1, that is women has Anemia, while the value is more than 12, the code is 0 (no Anemia). This is shown in column 3 of the Table 8. Now we fit a logistic regression between presence/absence of anemia and actual age using the software Jamovi. The following statistics is obtained (Table 10).

Predictor	Estimate	SE	t(Z)	Odds ratio	P	95% confidence interval
						Lower	Upper
Intercept	3.993	2.1239	1.88	54.216	0.060	0.844	3483.577
Age	−0.130	0.0629	−2.06	0.878	0.039	0.776	0.994

Table 10.

Model coefficients and 95% confidence intervals.

Since p value of regression coefficient (Age) is less than 0.05 and hence test is significant. That is, as age increases, chance of Anemia decreases.

11. Testing of hypothesis and confidence intervals of logistic regression coefficient

For testing the hypothesis that the sample comes from the population for which the value of logistic regression coefficient ${\beta }_1$ is equal to 0. That is, Our Null hypothesis $H_0:{\beta }_1=0$ against the alternate hypothesis $H_1:{\beta }_1\ne 0$.

Under the null hypothesis, for testing $H_0:{\beta }_1=0$, we define the test statistics given by

$$\mathrm{Z}=\frac{\widehat{{\beta }_1}-E\left(\widehat{{\beta }_1}\right)}{\mathit{SE}\left(\widehat{{\beta }_1}\right)}=\frac{\widehat{{\beta }_1}-{\beta }_1}{\mathit{SE}\left(\widehat{{\beta }_1}\right)}=\frac{\widehat{{\beta }_1}-0}{\mathit{SE}(\widehat{{\beta }_1}}=\frac{\widehat{{\beta }_1}}{\mathit{SE}(\widehat{{\beta }_1)}}=\frac{-.130}{0.0629}=-2.06.$$E66

We can obtain the 95% confidence interval of the logistic regression coefficient ${\beta }_1$as

Confidence interval for ${\beta }_1$= $e^{\widehat{{\beta }_1}\pm \mathrm{Z}\mathrm{SE}\left(\widehat{{\beta }_1}\right)}$. That is, lower limit = $e^{\widehat{{\beta }_1}-\mathrm{Z}\mathrm{SE}\left(\widehat{{\beta }_1}\right)}$ and upper limit = $e^{\widehat{{\beta }_1}+\mathrm{Z}\mathrm{SE}\left(\widehat{{\beta }_1}\right)}$.

For Example 9, lower limit = $e^{-0.130–1.96\times .0629}$ = 0.776 and upper limit = $e^{-0.130+1.96\times .0629}$= 0.993.

12. Conclusions

The main objective of this chapter is to discuss about simple linear regression and logistic regression analysis. Here, we have explained how to estimate regression coefficients, its standard error, testing of hypothesis of regression coefficients, 95% confidence intervals for simple linear regression and logistic regression model. All the statistics calculated manually is verified using R studio package and Jamovi package.

Acknowledgments

Author is thankful to the management committee members of the IntechOpen, the referee and the editor of this edited book for providing me an opportunity to share my works.