Hessian Approximation

Question 1

So if I had a large language model with 1 trillion parameters, how many entries would the Hessian have?

Answer 1

1 trillion by 1 trillion = 10^24 = one septillion

Question 2

How much memory would you need to store 1 septillion 2byte params?

Answer 2

2 yottabytes