A serious paper on bits as Joules per Kelvin

My ramblings about thermodynamics aren't so off-base, it turns out!

Remember this one? From 2009? Where I explained how Joules per Kelvin (energy per unit temperature) is a valid measure of information (or entropy, effectively "missing information"), which is normally measured in bits?

Well, now there's a paper that formalizes that idea and related ones. As the title ("Temperature as Joules per Bit") indicates, it looks at a rearranged version of the same insight (mine was "bits as Joules per temperature"). But, it also goes a lot deeper and derives thermodynamics starting from entropy to understand temperature, rather than the other way around, as is conventionally done.

Related thought: I remember back in that 2009 thermodyanmics/info theory frenzy, one of my goals was rederive the Carnot limit based on information-theoretic considerations -- that is, show it as a simple implication of the amount of knowledge you have about a system in a case where only know the temperature difference. (Naturally, I assumed someone had already done this and tried to find it but it was very hard to google for.)

Background: The Carnot limit tells you the maximum amount of mechanical work ("useful energy") you can extract from heat -- like, through an engine -- and, as it turns out, it's a function of the ratio of absoute temperatures you're working between. You don't face this limit when extracting work from a flywheel (spinning disc with grooves). Inspired by an counterintuitive insight in an Eliezer Yudkowsky LessWrong post, and my thoughts about it, I figured you could draw a more direct line from "knowledge of a temperature difference" to "how much energy is extractable".

Now I'll give it a go with ChatGPT, and post my findings!

When you can't go back to sleep: thermodynamics

Did you miss my posting? Well, it's been one of those days when you wake up early and can't go back to sleep. My mind's running wild this morning, and I figured I'd make some good use out of it. (It is no longer early as of completing this post because of interruptions from a playful kitty.)

I'm going to continue the lesson about thermodynamics that last left off about a year ago, by discussing some more interesting implications of the idea that "energy per unit temperature" is a measure of degrees of freedom. You see, in the time since then, I read John S. Avery's book Information Theory and Evolution, which, as you might have inferred, discusses life from the perspective of that ever-so-useful field of information theory. He also applies it to cultural (often called "memetic") evolution.

The first interesting insight that this book alerted me to is about molar entropy. Some background: in your chemistry class, you might have learned about the Gibbs free energy of a reaction, ΔG, which is calculated from ΔH - TΔS, where H is the molar enthalpy (internal + flow energy per mole), T is absolute temperature, and S is the molar entropy. For a chemical reaction, you look up the molar enthalpies of the products and subtract off the enthalpies of the reactants. Then you do the same for molar entropies, multiplying by the absolute temperature at which the reaction takes place, and add them. A negative sign for ΔG means the reaction happens spontaneously (well, as long as there is an available pathway).

With that out of the way, what are the units for S? Most tables give them as J/K*mol (Joules per Kelvin per mole, or energy per unit temperature per quantity of molecules). But, as the last post in this series showed, energy per unit temperature measures degrees of freedom, which can also be expressed in bits. So, as Avery neatly derives on pages 81-82, you can also express molar entropy in bits per molecule. (The conversion factor is 1 J/K*mol = 0.1735 bits/molecule.) I find this a much more intuitive way to think about it, because it connects the concept of molar entropy to the underlying dynamic: how many bits of information (on average) do you need to specify a molecule's current state, beyond that which you know from the temperature?

Also, rather than having to empirically derive this value directly (either from reaction data or by integrating its specific heat capacity per unit temperature from 0 K to its current temperature), it can be inferred from the known properties of the molecule: its shape, size, and bond strength. The stronger ("stiffer") its bonds are, the lower the entropy of the molecule, because large deviations from its equilibrium configuration are less probable. (Diamond, with its very strong covalent bonds, has the incredibly low molar entropy of 0.24 bits per carbon atom at STP, meaning you need less than one bit of information to specify every four atoms.)

[ADDENDUM: Avery also adds that if you divide the Gibbs equation through by T, you can describe a reaction in terms of the "information lost", i.e., the greater number of degrees of freedom you have permitted by letting the reaction take place.]

By recognizing this interconnection between molecule properties and complexity (needing more information to fully specify = more complex), one sees more unity ("consilience") to the science as a whole: entropy and bond properties aren't just off in their own domains, but have a lawful relationship. Unfortunately, however, I haven't worked out how to derive entropy from stiffness of a degree of freedom, and I haven't found a text that does it either.

Next in the series: A discussion of Eric J. Chaisson's Cosmic Evolution: The Rise of Complexity in Nature, which proposes specific energy flux (energy flow through a system per unit mass) as a measure of complexity that is applicable to everything from stars to planets to life to vehicles to computer chips to culture.

Mixing economics, thermodynamics, and heterogeneity

... or METH, as some call it. And if you want some more drug innuendo, read on.

On Brad DeLong's blog, a commentator named "MJ" deftly applies insights from thermodynamics to the issue of heterogeneity of goods in economics:

What would statistical mechanics be without a quantitative model of heterogeneous vs. homogeneous distributions? Such a statistical mechanics would miss a few subtle but crucial concepts. Such as entropy.

Note how the negentropic development of increasingly heterogeneous capital allocations over the past decade was accomplished through entropy production: bundling good with bad, compromising tranches, etc... Goldman Sachs made a killing, basically, off of knowingly producing entropy. The entropy production, of course (2nd law), far exceeded the negentropy production of their wealth aggregation- as reflected in the order of magnitude between financial industry's gains and the over all loss.

We're now learning Fannie and Freddie also engaged in entropy production, obscuring the distinction between scores over and under 660.

I can vouch for that as showing a good understanding of entropy, and it gives a good perspective for viewing economics:

1) An efficient economy produces as little net entropy as possible: the entropy it generates (destruction of heterogeneity) should be offset by the entropy it destroys in organizing inputs for their uniquely optimal roles.

2) A sign of inefficiency is when economic actors destroy distinctions (like in MJ's example of very different tranches and borrowers being made indistinguishable) without making a corresponding useful distinction or organization.

Definitely some issues worth fleshing out. I know I've seen papers that try to view economics from a thermodynamic perspective, but they invariably have me rolling my eyes.

Setting Callahan Straight -- on reductionism and the history of science

In a recent post on his blog, Gene Callahan tried to defend the "mysterious vital force" hypothesis from the clueless reductionists, saying that hey, it was a good idea at the time, and might even show some promise today.

After I explained that the "vital force" of the 18th century used to "explain" life (in the way that the "train force" explains the motion of trains) was not scientific in the way that gravitational force was, the exchange started to get lengthy. So, he did was all pursuers of the Truth do: he closed comments.

And this isn't the first time for Callahan to use argumentum ad closum: in a previous exchange where "TokyoTom" was the critic, he did the same thing, and TokyoTom recounts the pre-cutoff (and pre-coverup!) exchange on his blog.

So, since I can't post on Callahan's blog for that discussion, I decided to post my response to his latest comment here. So, take a gander at the discussion thus far, and read my response below. Let's hope it's just a "blog malfunction" this time, too!

********

Now, of course, this is not an explanation, just like Newton's force laws do not "explain anything," as the Cartesians ceaselessly pointed out.

And the Cartesians were dead wrong to argue this. Generating a model that correctly predicts the paths of celestial bodies (and bodies in vacuums, and falling bodies on earch with a high mass/drag ratio, etc.) is an explanation. The alternative -- your alternative -- results in such absurdities as "Okay, okay, sure, you were able to accurately predict celestial motions, material strengths, rocket propulsion capabilities, air properties, blah blah blah ... but you don't truly understand what it means to fly to the moon and return safely. Yeah, you have a 'model' that accurately predicts all that stuff, but you didn't truly [*suggestive emphasis*] explain it."

What else could you want from an explanation? The satisfaction of Gene Callahan's personal aesthetic standards? Or your presuppositions about things that have to be there?

Descartes and other mechanists developed models showing how gravity and magnetism could be reduced to the motion of particles alone -- Descartes, for instance, posited a flow of little screw shaped particles drawing iron to magnets.

And that would be a *different* explanation. It may be a better or worse explanation. But "reducing to particle motions" is not a requirement for an explanation, especially if it's wrong or unnecessarily long. Newton's model remains an explanation in that it constrains our expectations.

At the time, vitalism was a sound scientific hypothesis, and the vitalists expected to find good force laws just like for gravity, etc.

And they had no basis for such an expectation, even before an experiment, since "the vital force" is an "explanation" for everything, and therefore nothing. No matter what experiment they performed, it would be consistent with a "vital force". That was how they actually used the term.

me:"Your claim was stronger because you are saying that now, even given all the scientific knowledge we have, there still remains hope for the vital force "hypothesis"..."

you:All sorts of scientific hypotheses get revived after being "soundly rousted" earlier.

Right, but about the issue we were discussing: My claim, the one you're responding to here, is that the position you do endorse ("Vitalism may turn out to be a fruitful hypothesis") is a stronger claim than the one you excused earlier vitalists for believing (that it's reasonable given the limited evidence so far).

Yes, theories do get revived after being discarded. But as for the point that I was actually discussing there, it's irrelevant; the fact remains that you are making a stronger claim.

Now, with that said, scientists are indeed looking for causal phenomena and more general laws they may not have noticed before. But they keep a *broad* outlook. There are many, many shorter hypotheses to rule out before positing an additional, ontologically-fundamental "vital force" -- if in fact the "hypothesis" claims anything at all.

Oh, and what do you think about the complete overthrow of reductionism in quantum mechanics, where the behaviour of "atomic" particles turns out to be not atomic at all, but dependent instantaneously on the state of all other particles they have interacted with throughout their entire history, so that ultimately their state depends on the state of the entire universe?

Now you're just showing more ignorance of science on top of ignorance of reductionism. It was the philosophers, not physicists, who fell into the trap of thinking of particles, rather than an amplitude distribution over configurations, as being ontologically fundamental entities, and so were seduced to believe, "hah, I can say that you can never prove two electrons are identical, because that's an issue of epistemology, which I'm an expert in, rather than physics", which is wrong.

Reductionism of the kind I (and Drescher and Pearl and Jaynes and Hofstadter) endorse was never predicated on the existence of "atomic particles". Rather, it identifies where such hidden assumptions come in.

What modern quantum physics shows is that the wave equation is deterministic, and each point need only look at its immediate neighbors to iterate to the next state. See the excellent work of Gary Drescher in Good and Real: Demystifying Paradoxes from Physics to Ethics, or, for an online source, Eliezer Yudkowsky's quantum physics sequence.

Another interesting thermodynamics result

Here's another interesting insight on thermodynamics and information theory to add to my previous: I realized why "joules per kelvin" is a measure of entropy. Not exciting? Wait, you'll see.

In the previous post on this topic, I mentioned all the parallels between entropy in information theory and entropy in thermodynamics. Also, some properties can be calculated by their information-theoretic definition or their thermodynamic definition, such as the thermodynamic availability, which can be calculated as the Kullback-Leibler divergence, a measure from information theory. But what's interesting is that this value can be expressed in terms of bits, or in terms of Joules per Kelvin, which has units of energy over temperature, with a simple constant multiplier for conversion.

Huh?

You see, there's the hard part: why on earth would bits -- which measure how much memory your computer has -- possibly refer to the same property as "Joules per Kelvin", the way that inches and meters refer to the same property?

And that's where we get to the interesting part. First of all, what is temperature? It's not how much internal energy something has, but rather, it's internal energy per degree of freedom. In this context, a "degree of freedom" is a distinct way that something can be modified at the molecular level. A single-atom molecule may be viewed as having three degrees of freedom, since it can translate in three dimensions. Once the molecule has shape, however, it can rotate in addition to translating. So, two different substances at the same temperature can have different internal energy, because one of them may be stuffing that energy into more degrees of freedom.

So where does that get us with Joules per Kelvin and energy per unit temperature? Well, watch what happens when you expand out temperature in the entropy expression:

energy
------------------------
energy/degree-of-freedom

= energy * degree-of-freedom/energy

= degree-of-freedom (!)

So there you have it! Once you expand it out, energy per unit temperature is simply a roundabout way of saying "degrees of freedom".

Now you may ask, "Nice, but that still doesn't explain what that has to do with bits." But then, what is a bit but a binary degree of freedom? When you have memory of n bits, then there are n values that you can independently set to one of two possible values, making it likewise a measure of degrees of freedom. (Note that this capability allows you to store 2^n possible states.) And informational entropy, in turn -- also expressed in bits -- is the logarithm of the number of possible states a system can be in, making it proportional to the degrees of freedom as well.

The two lessons to take away are that:

1) The number of degrees of freedom a system has depends on the arbitrary choice of what you count as a degree of freedom, just like the number of "units of length" something is.

2) Whichever consistent method you use of counting degrees of freedom, the number of degrees of freedom is proportional to the logarithm of the number of possible states.

Mystery solved! (No, I don't know if this discussion is given in any textbook treatment of the issue.)

Oh, and: Happy Saint Patty's Day!

My plan to destroy the universe won't work

And I'll bet you're relieved!

Maybe a little background is in order.

A question of interest to philosophers and theoretical physicists is whether or not the universe is just a simulation running on some computer, one level up. (See e.g. Nick Bostrom's Simulation Argument.) Of course, many ridicule this idea as being non-falsifiable and thus non-scientific.

Not so fast! I said. Of course it's falsifiable. Here's how: if the universe is a simulation, then its programmers probably try to economize on computational resources (computing cycles, memory, disc space, time, etc). And to do that, they will make the program reveal to "us" (the conscious entities) the minimum required to make everything appear "believable". That in turn, means that as long as we "wouldn't know the difference" if some physical process developed in a way contradicting the rest of our observations, the simulator won't bother to churn through the calculations needed to make the process match up with known universal laws. In other words: "If we're not looking, why bother making sure something's there?"

And that tells us how to test the Simulation Hypothesis: have everyone set up as much measurement equipment as they can, and therefore observe as much as they can. This will force the simulator do many more calculations than it would otherwise have to, since now it has to keep consistent with that many more observations. The programmers then have to devote an ever-increasing amount of resources to keep it running, which will eventually force them to "cut corners" in implementing the laws of physics, revealing violation of Standard Model physics, or ... um, make them pull the plug on our existence.

Hence, my "plan to destroy the universe".

Now, the good news: the plan wouldn't work, based on what we already know about how the universe would react to such a "hypermeasurement" scenario! And the reason is shocking: because we can't actually increase our total knowledge.

"What in the hay-ll? I did me some book-larnin' not but three yurs ago!"

Sorry, that was Cletus, our resident country bumpkin.

Well, I'll need to some more background now to justify that claim. First, I want to point you to a post on OvercomingBias.com that introduced me to a lot about what I'll discuss here: Engines of Cognition.

Now, consider the 2nd law of thermodynamics. There are many ways to express it, but a simpler way is: "The amount of disorder ('entropy') in the universe must always increase." Sure, you can increase the order any one specific place -- say, when you form crystals -- but it will always be counterbalanced by an increase in disorder somewhere else. The most common application of this law is in heat engines (such as the one in your car): when you burn fuel to turn your engine and thus your tires, you are extracting a kind of order: the useful mechanical "work" (as it is called in physics) of a spinning engine. However, to do so, you burn fuel and transfer heat to the environment, which, when tabulated, generates entropy/disorder exceeding that which you destroyed in extracting mechanical work from the system to drive.

Now, here's the kicker: there are deep parallels between the concept of entropy in thermodynamics, and the concept called "entropy" in information theory. In the latter, it refers (roughly) to the uncertainty one has about the content of a message before reading it. Any knowledge that some kinds of messages are more likely than others therefore reduces that "entropy". Similarly, entropy is at a maximum when all messages are equally likely.

And the truly mind-blowing part is that the connection between the two kinds of entropy is so deep that entropy in the information-theoretic sense affects entropy in the thermodynamic sense. (This is going somewhere, just be patient.) In short, if you are able to reduce your uncertainty (information-theoretic entropy) about the "message" contained in the molecules of a system, that knowledge can actually be exploited to reduce the thermodynamic entropy of the system and thereby extract useful work! (For reference, and early exploration of this idea is called the Maxwell's Demon thought experiment, and a hypothetical engine that extracts work this way is the Szilard engine.)

But this hypothetical capability of decreasing the entropy of a system does not actually contradict the 2nd Law, which, you'll remember, says that total entropy must increase. Rather, for reasons I won't go into, this acquisition of knowledge itself is limited by the 2nd Law. Just as the extraction of "organized" mechanical work from fuel requires the generation somewhere else, of at least as much counterbalancing disorganization, so too does the collection of information that could permit extraction of the same work without the fuel require a counterbalancing loss of information somewhere else, i.e. increased uncertainty.

This principle reveals a fundamental limit that your brain (in a deep sense, a "cognitive engine") faces: in order to learn something true about your environment (whether via the senses or inferences), you must sacrifice knowledge somewhere else. Fortunately, nothing requires you to care much about that lost knowledge, which takes the form of "lost certainty about aggregate statistical properties of thermodynamic variables".

Now, back to the main point: from the perspective of hypothetical beings running the universe's simulator, my idea to gather more measurements has no impact. Any time we make a measurement, we are gathering knowledge, which must therefore correspond to lost knowledge somewhere else. So, far from threatening the computer's ability to simulate our universe, all our measurements will (amazingly) decrease the computational resources the simulator requires.

Which neatly returns the Simulation Hypothesis to non-falsifiability, and assures us that even if people acted on my idea, we're still safe and sound. Alternatively, it reveals the universe's programmers to be really, really clever :-)