I am putting a lot of data into Microsoft Excel that needs to be squared and totaled for a scientific experiment. However, the numbers I get from Excel differ significantly from the numbers I get from a graphing calculator and the scientific calculator in my computer.

 

When I try 0.18^2 in Excel I get 0.0316 but I get 0.0324 on a calculator. When I add all the numbers together, they aren't even close. I've tried changing the number of decimal places in an Excel cell, but that doesn't make a difference. Does anyone have any suggestions?

Check what number system your calculator is using.

use open office? never seen any mathematical discretions with that

use a proper statistical program like R or origin.

I tried Open Office, but I had the exact same problem. I know my calculator is correct because those numbers match my teacher's answers in the sample experiment he provided.

 

It works when I only cross-reference one cell at a time. For example, I can say cell A1=0.18 and cell B1=A1^2 and then I will get the answer 0.0324 in B1. However, there is a lot of data and I have cells that cross-reference other cells with equations.

 

For example, cell A1 has the number .5 and the average of all numbers in column A is .68. Cell B1 has the equation A1-.68 which is -0.18. Cell C1 has the equation B1^2 which is giving the incorrect answer of 0.0316.

what if you change it to B1 * B1?

I tried that, but I still got the same incorrect answer.

Well without seeing what you're doing I can offer little help.

You can visit http://eflierz.com/Ttest.xls to download the spreadsheet I'm working on.

 

Also, here is a screenshot of it. Thank you very much for any help you can provide.

 

Yep I see you're problem streight away. It's not doing 0.11^2

 

Have a look at the number of decimal places showing in cell D5.

Hi Klaynos,

 

Thank you for your reply. I looked at the decimal places, and I see 2 after the decimal. I'm not very good at math, but should I change this? My professor wanted column D to have 2 decimal places and column E to have 4. On his sample answers, cell D5 reads "0.11" and cell E5 reads "0.0121".

Well, you need to enforce rounding if you want it to be identicle to his.

 

At the moment your D5, displays the value 0.11, but it's actual values is:

 

0.108333333333333000

 

Which is the value it uses for all it's calculations. And it might only become 0 after that point because of a limitation in the presicion of openoffice.org which is what I used to open it.