A serious paper on bits as Joules per Kelvin

My ramblings about thermodynamics aren't so off-base, it turns out!

Remember this one? From 2009? Where I explained how Joules per Kelvin (energy per unit temperature) is a valid measure of information (or entropy, effectively "missing information"), which is normally measured in bits?

Well, now there's a paper that formalizes that idea and related ones. As the title ("Temperature as Joules per Bit") indicates, it looks at a rearranged version of the same insight (mine was "bits as Joules per temperature"). But, it also goes a lot deeper and derives thermodynamics starting from entropy to understand temperature, rather than the other way around, as is conventionally done.

Related thought: I remember back in that 2009 thermodyanmics/info theory frenzy, one of my goals was rederive the Carnot limit based on information-theoretic considerations -- that is, show it as a simple implication of the amount of knowledge you have about a system in a case where only know the temperature difference. (Naturally, I assumed someone had already done this and tried to find it but it was very hard to google for.)

Background: The Carnot limit tells you the maximum amount of mechanical work ("useful energy") you can extract from heat -- like, through an engine -- and, as it turns out, it's a function of the ratio of absoute temperatures you're working between. You don't face this limit when extracting work from a flywheel (spinning disc with grooves). Inspired by an counterintuitive insight in an Eliezer Yudkowsky LessWrong post, and my thoughts about it, I figured you could draw a more direct line from "knowledge of a temperature difference" to "how much energy is extractable".

Now I'll give it a go with ChatGPT, and post my findings!

ChatGPT4 refines my argument about fish being bad swimming instructors

I finally splurged for a paid ChatGPT subscription! With access to GPT-4, I asked it to refine my earlier argument about fish being bad swimming instructors. Here's what it gave me:

***

Certainly, let's rework the argument using more scientifically grounded explanations and references:

While it's true that fish, with their adept aquatic abilities, may seem like logical swimming instructors for humans at first glance, upon closer examination, it becomes clear that this is far from practical or effective. Even if we put aside the substantial communication and physical barriers, there are fundamental biological and behavioral differences between fish and humans that pose significant challenges.

Humans are terrestrial, bipedal mammals, and our motor skills and locomotive mechanisms are naturally adapted to this lifestyle. Fish, in contrast, are aquatic creatures designed to maneuver efficiently in water (Webb, P.W., 1982, "Locomotor patterns in the evolution of actinopterygian fishes"). Their ability to swim is an innate behavior, hardwired into their neural circuitry, not a learned skill (Gould, J.L., 1982, "Ethology: The Mechanisms and Evolution of Behavior"). Given this, fish cannot consciously comprehend or articulate the mechanics of their swimming abilities as humans would require in a teaching-learning scenario.

In the field of pedagogy, it is well accepted that the process of teaching involves more than simply demonstrating a skill; it requires an understanding of learning styles, the ability to give and receive feedback, and a degree of self-awareness in the teacher (Boud, D., et al., 1985, "Reflection: Turning Experience into Learning"). Fish, as far as current scientific consensus suggests, lack these cognitive capacities (Butler, A.B., et al., 2005, "Evolution of the Neural Basis of Consciousness: A Bird-Mammal Comparison").

Furthermore, many human skills, including swimming, are learned through a combination of instruction, practice, and feedback (Kirschner, P.A., et al., 2006, "Why Minimal Guidance During Instruction Does Not Work"). This model of learning doesn't align with the unconscious, reflex-driven behaviors seen in fish.

Lastly, it's important to acknowledge the vast physiological differences between humans and fish. Human limbs and fish fins have evolved for different purposes and environments, with humans developing more for grasping and manipulation and fish for efficient aquatic locomotion (Thorsen, D.H., et al., 2004, "Biomechanics of Locomotion in Asian Carp"). These differences are fundamental and cannot be bridged by simple observation or imitation.

In conclusion, despite their swimming prowess, fish cannot be effective swimming instructors for humans due to biological, behavioral, and cognitive differences, as well as practical communication and anatomical obstacles. This argument does not discount the value of observing aquatic creatures to improve our understanding of efficient movement in water, but it's clear that such observation is a far cry from active instruction.

Another parallel between IP and EM spectrum rights

I constantly bring up the EM spectrum in discussions of IP, and for good reason. It's quite difficult to justify rights in one but not the other, leading some people to unpalatable conclusions. Heck, even Stephan Kinsella, the big anti-IP poomba, isn't sure whether there should be rights to radio waves.

But recently, I've found yet another parallel. Consider the case of someone who, like in lots of instructive thought experiments, wants to block the propagation of radio waves from a nearby tower so that people can't receive them (well, extract information from them, but you get the point).

There's a device that lets you block EM waves, known as a Faraday cage. I didn't know how they work before, but it turns out that, by being made of conducting material, their electrons realign so as to produce the *opposite* field from the one around it (which superposes over it and cancels it out).

See where this is going? To protect your right to to transmit information via EM waves, you need to be able to prevent others from ... er ... instantiating the same pattern! Now, where else have people asserted that kind of right?

(By the way, first post ever from my first smartphone, the wonderful Samsung Moment. And no, unlike when Tyler Cowen promotes a product, I didn't get it for free or otherwise get paid to say that. No, the links weren't added from the Moment.)

Silas Barta, information theorist by night

UPDATE 12/17/09: Steven Landsburg, after responding several times in the comments section here, posts a defense of his position on his blog, although without mentioning me or Bob Murphy. Hey, I can understand: if I were in his position, I'd hide the existence of me and Bob too!

***

Bob Murphy invokes my expertise on information theory to criticize (yet) another bizarre argument from Steven Landsburg, that the natural numbers are more complex than human life. Here's the mistaken part of Landsburg's reasoning:

...the most complex thing I’m aware of is the system of natural numbers (0,1,2,3, and all the rest of them) together with the laws of arithmetic ...

If you doubt the complexity of the natural numbers, take note that you can use just a small part of them to encode the entire human genome. That makes the natural numbers more complex than human life. Unless, of course, human beings contain an uncodable essence, like an immortal soul

Naturally, I don't necessarily agree with the broader theological points Bob makes in his reply, and such issues will remain even scarcer on this blog than on his. However, I will expand on point I made in discussion with Bob.

The error in Landsburg's line of reasoning is: the fact that you can use instances of X to build Y does not mean X is more complex than Y. Just the opposite, in fact: in order to describe Y, you must describe X as a substep. Like in the analogy I gave, you can use bricks and mortar to build a house, but that means it's the house that's more complex. To fully specify the house you must describe not only the bricks and mortar, but the form they take as a house -- how they're supposed to be put together.

As for arithmetic and natural numbers, it's their lack of complexity that makes them so useful. By appealing to it, you can make sense of a diverse array of phenomena. The more complex arithmetic were, the less helpful it would be in making sense of things.

Just to be clear, this doesn't mean it's easy to learn math (different people have different problems in different topics and levels), or that you can't do anything complex with math. The point is that no amount of complexity produced in using arithmetic could ever imply arithmetic's complexity, for the same reason that no matter how complex a house you make with one kind of brick, you can't make the brick more complex.

But of course, Landsburg's errors don't end there. He wants to go so far as to say that by merely encoding the genome in base 4, you've described human life. That's certainly the impression people get from discussions of DNA in the popular media and movies like Jurassic Park. Hey, all you need is a string of letters made up of A,G,C,T, and you've described someone completely!

To put it mildly: that's not how it works. First of all, you need to say what the letters actually mean. And then, even if you know that much, all you have are empty labels -- suggestively named LISP tokens. So you know that C is cytosine? Okay, but what's that? Now you need to describe where the carbons and nitrogens and oxygens go to make up cytosine. But wait -- what's this "nitrogen" thing, anyway? And so on.

Don't worry -- the process terminates: once you've described the generative model that puts all of these concepts together in a way that yields a description of human life as its output.

Needless to say, you're using more than a few integers by that point!

World's newest space agency: Reuters

I normally don't pay much attention to photo credits, but I had to do a double-take on this one. An article in the Telegraph has a satellite picture of the sun. Of course, to get that kind of picture, you have to get pretty close, exist in a high temperature environment, and have photography equipment capable of significantly attenuating the EM radiation thrown off from the sun

And who do they credit for the photo? NASA, right? No, we get:

Professor Henrik Svensmark argued that the recent warming period was caused by solar activity. Photo: REUTERS

Um, yeah dude. I think Reuters got that photo from someone else. With the budget cuts the media have had to make in the past few years, they can only afford near-earth satellites. Deeper-space probes are just out of the question.

ANYWAY, since I haven't posted on Climategate, or for that matter, anything in a while, here are my thoughts: It's absolutely disgraceful, the way the scientists in question have acted. Disclosure of your data does not mean that skeptics get to go on a multi-year scavenger hunt to find your raw data and then play guessing games about which sources you threw out and why.

The very fact that you have to make a post like this one in order to summon forth all the data is proof that you weren't being transparent enough.

There's also clear evidence that the scientists didn't seem to understand that you can't contort one data source to look like another and then call it two independent sources of data. Eric S. Raymond has done a tremendous job at exposing the tricks in the code, which explains exactly why the insular climate science doesn't want critics poring over their work

Oh, and just a hint: when you only allow people you approve of to review your work, that's not science.

PS: Recall that my outrage at many libertarians has been to their reactions *conditional* on AGW being real, and that outrage remains.

Setting Callahan Straight -- on reductionism and the history of science

In a recent post on his blog, Gene Callahan tried to defend the "mysterious vital force" hypothesis from the clueless reductionists, saying that hey, it was a good idea at the time, and might even show some promise today.

After I explained that the "vital force" of the 18th century used to "explain" life (in the way that the "train force" explains the motion of trains) was not scientific in the way that gravitational force was, the exchange started to get lengthy. So, he did was all pursuers of the Truth do: he closed comments.

And this isn't the first time for Callahan to use argumentum ad closum: in a previous exchange where "TokyoTom" was the critic, he did the same thing, and TokyoTom recounts the pre-cutoff (and pre-coverup!) exchange on his blog.

So, since I can't post on Callahan's blog for that discussion, I decided to post my response to his latest comment here. So, take a gander at the discussion thus far, and read my response below. Let's hope it's just a "blog malfunction" this time, too!

********

Now, of course, this is not an explanation, just like Newton's force laws do not "explain anything," as the Cartesians ceaselessly pointed out.

And the Cartesians were dead wrong to argue this. Generating a model that correctly predicts the paths of celestial bodies (and bodies in vacuums, and falling bodies on earch with a high mass/drag ratio, etc.) is an explanation. The alternative -- your alternative -- results in such absurdities as "Okay, okay, sure, you were able to accurately predict celestial motions, material strengths, rocket propulsion capabilities, air properties, blah blah blah ... but you don't truly understand what it means to fly to the moon and return safely. Yeah, you have a 'model' that accurately predicts all that stuff, but you didn't truly [*suggestive emphasis*] explain it."

What else could you want from an explanation? The satisfaction of Gene Callahan's personal aesthetic standards? Or your presuppositions about things that have to be there?

Descartes and other mechanists developed models showing how gravity and magnetism could be reduced to the motion of particles alone -- Descartes, for instance, posited a flow of little screw shaped particles drawing iron to magnets.

And that would be a *different* explanation. It may be a better or worse explanation. But "reducing to particle motions" is not a requirement for an explanation, especially if it's wrong or unnecessarily long. Newton's model remains an explanation in that it constrains our expectations.

At the time, vitalism was a sound scientific hypothesis, and the vitalists expected to find good force laws just like for gravity, etc.

And they had no basis for such an expectation, even before an experiment, since "the vital force" is an "explanation" for everything, and therefore nothing. No matter what experiment they performed, it would be consistent with a "vital force". That was how they actually used the term.

me:"Your claim was stronger because you are saying that now, even given all the scientific knowledge we have, there still remains hope for the vital force "hypothesis"..."

you:All sorts of scientific hypotheses get revived after being "soundly rousted" earlier.

Right, but about the issue we were discussing: My claim, the one you're responding to here, is that the position you do endorse ("Vitalism may turn out to be a fruitful hypothesis") is a stronger claim than the one you excused earlier vitalists for believing (that it's reasonable given the limited evidence so far).

Yes, theories do get revived after being discarded. But as for the point that I was actually discussing there, it's irrelevant; the fact remains that you are making a stronger claim.

Now, with that said, scientists are indeed looking for causal phenomena and more general laws they may not have noticed before. But they keep a *broad* outlook. There are many, many shorter hypotheses to rule out before positing an additional, ontologically-fundamental "vital force" -- if in fact the "hypothesis" claims anything at all.

Oh, and what do you think about the complete overthrow of reductionism in quantum mechanics, where the behaviour of "atomic" particles turns out to be not atomic at all, but dependent instantaneously on the state of all other particles they have interacted with throughout their entire history, so that ultimately their state depends on the state of the entire universe?

Now you're just showing more ignorance of science on top of ignorance of reductionism. It was the philosophers, not physicists, who fell into the trap of thinking of particles, rather than an amplitude distribution over configurations, as being ontologically fundamental entities, and so were seduced to believe, "hah, I can say that you can never prove two electrons are identical, because that's an issue of epistemology, which I'm an expert in, rather than physics", which is wrong.

Reductionism of the kind I (and Drescher and Pearl and Jaynes and Hofstadter) endorse was never predicated on the existence of "atomic particles". Rather, it identifies where such hidden assumptions come in.

What modern quantum physics shows is that the wave equation is deterministic, and each point need only look at its immediate neighbors to iterate to the next state. See the excellent work of Gary Drescher in Good and Real: Demystifying Paradoxes from Physics to Ethics, or, for an online source, Eliezer Yudkowsky's quantum physics sequence.

Another interesting thermodynamics result

Here's another interesting insight on thermodynamics and information theory to add to my previous: I realized why "joules per kelvin" is a measure of entropy. Not exciting? Wait, you'll see.

In the previous post on this topic, I mentioned all the parallels between entropy in information theory and entropy in thermodynamics. Also, some properties can be calculated by their information-theoretic definition or their thermodynamic definition, such as the thermodynamic availability, which can be calculated as the Kullback-Leibler divergence, a measure from information theory. But what's interesting is that this value can be expressed in terms of bits, or in terms of Joules per Kelvin, which has units of energy over temperature, with a simple constant multiplier for conversion.

Huh?

You see, there's the hard part: why on earth would bits -- which measure how much memory your computer has -- possibly refer to the same property as "Joules per Kelvin", the way that inches and meters refer to the same property?

And that's where we get to the interesting part. First of all, what is temperature? It's not how much internal energy something has, but rather, it's internal energy per degree of freedom. In this context, a "degree of freedom" is a distinct way that something can be modified at the molecular level. A single-atom molecule may be viewed as having three degrees of freedom, since it can translate in three dimensions. Once the molecule has shape, however, it can rotate in addition to translating. So, two different substances at the same temperature can have different internal energy, because one of them may be stuffing that energy into more degrees of freedom.

So where does that get us with Joules per Kelvin and energy per unit temperature? Well, watch what happens when you expand out temperature in the entropy expression:

energy
------------------------
energy/degree-of-freedom

= energy * degree-of-freedom/energy

= degree-of-freedom (!)

So there you have it! Once you expand it out, energy per unit temperature is simply a roundabout way of saying "degrees of freedom".

Now you may ask, "Nice, but that still doesn't explain what that has to do with bits." But then, what is a bit but a binary degree of freedom? When you have memory of n bits, then there are n values that you can independently set to one of two possible values, making it likewise a measure of degrees of freedom. (Note that this capability allows you to store 2^n possible states.) And informational entropy, in turn -- also expressed in bits -- is the logarithm of the number of possible states a system can be in, making it proportional to the degrees of freedom as well.

The two lessons to take away are that:

1) The number of degrees of freedom a system has depends on the arbitrary choice of what you count as a degree of freedom, just like the number of "units of length" something is.

2) Whichever consistent method you use of counting degrees of freedom, the number of degrees of freedom is proportional to the logarithm of the number of possible states.

Mystery solved! (No, I don't know if this discussion is given in any textbook treatment of the issue.)

Oh, and: Happy Saint Patty's Day!

My plan to destroy the universe won't work

And I'll bet you're relieved!

Maybe a little background is in order.

A question of interest to philosophers and theoretical physicists is whether or not the universe is just a simulation running on some computer, one level up. (See e.g. Nick Bostrom's Simulation Argument.) Of course, many ridicule this idea as being non-falsifiable and thus non-scientific.

Not so fast! I said. Of course it's falsifiable. Here's how: if the universe is a simulation, then its programmers probably try to economize on computational resources (computing cycles, memory, disc space, time, etc). And to do that, they will make the program reveal to "us" (the conscious entities) the minimum required to make everything appear "believable". That in turn, means that as long as we "wouldn't know the difference" if some physical process developed in a way contradicting the rest of our observations, the simulator won't bother to churn through the calculations needed to make the process match up with known universal laws. In other words: "If we're not looking, why bother making sure something's there?"

And that tells us how to test the Simulation Hypothesis: have everyone set up as much measurement equipment as they can, and therefore observe as much as they can. This will force the simulator do many more calculations than it would otherwise have to, since now it has to keep consistent with that many more observations. The programmers then have to devote an ever-increasing amount of resources to keep it running, which will eventually force them to "cut corners" in implementing the laws of physics, revealing violation of Standard Model physics, or ... um, make them pull the plug on our existence.

Hence, my "plan to destroy the universe".

Now, the good news: the plan wouldn't work, based on what we already know about how the universe would react to such a "hypermeasurement" scenario! And the reason is shocking: because we can't actually increase our total knowledge.

"What in the hay-ll? I did me some book-larnin' not but three yurs ago!"

Sorry, that was Cletus, our resident country bumpkin.

Well, I'll need to some more background now to justify that claim. First, I want to point you to a post on OvercomingBias.com that introduced me to a lot about what I'll discuss here: Engines of Cognition.

Now, consider the 2nd law of thermodynamics. There are many ways to express it, but a simpler way is: "The amount of disorder ('entropy') in the universe must always increase." Sure, you can increase the order any one specific place -- say, when you form crystals -- but it will always be counterbalanced by an increase in disorder somewhere else. The most common application of this law is in heat engines (such as the one in your car): when you burn fuel to turn your engine and thus your tires, you are extracting a kind of order: the useful mechanical "work" (as it is called in physics) of a spinning engine. However, to do so, you burn fuel and transfer heat to the environment, which, when tabulated, generates entropy/disorder exceeding that which you destroyed in extracting mechanical work from the system to drive.

Now, here's the kicker: there are deep parallels between the concept of entropy in thermodynamics, and the concept called "entropy" in information theory. In the latter, it refers (roughly) to the uncertainty one has about the content of a message before reading it. Any knowledge that some kinds of messages are more likely than others therefore reduces that "entropy". Similarly, entropy is at a maximum when all messages are equally likely.

And the truly mind-blowing part is that the connection between the two kinds of entropy is so deep that entropy in the information-theoretic sense affects entropy in the thermodynamic sense. (This is going somewhere, just be patient.) In short, if you are able to reduce your uncertainty (information-theoretic entropy) about the "message" contained in the molecules of a system, that knowledge can actually be exploited to reduce the thermodynamic entropy of the system and thereby extract useful work! (For reference, and early exploration of this idea is called the Maxwell's Demon thought experiment, and a hypothetical engine that extracts work this way is the Szilard engine.)

But this hypothetical capability of decreasing the entropy of a system does not actually contradict the 2nd Law, which, you'll remember, says that total entropy must increase. Rather, for reasons I won't go into, this acquisition of knowledge itself is limited by the 2nd Law. Just as the extraction of "organized" mechanical work from fuel requires the generation somewhere else, of at least as much counterbalancing disorganization, so too does the collection of information that could permit extraction of the same work without the fuel require a counterbalancing loss of information somewhere else, i.e. increased uncertainty.

This principle reveals a fundamental limit that your brain (in a deep sense, a "cognitive engine") faces: in order to learn something true about your environment (whether via the senses or inferences), you must sacrifice knowledge somewhere else. Fortunately, nothing requires you to care much about that lost knowledge, which takes the form of "lost certainty about aggregate statistical properties of thermodynamic variables".

Now, back to the main point: from the perspective of hypothetical beings running the universe's simulator, my idea to gather more measurements has no impact. Any time we make a measurement, we are gathering knowledge, which must therefore correspond to lost knowledge somewhere else. So, far from threatening the computer's ability to simulate our universe, all our measurements will (amazingly) decrease the computational resources the simulator requires.

Which neatly returns the Simulation Hypothesis to non-falsifiability, and assures us that even if people acted on my idea, we're still safe and sound. Alternatively, it reveals the universe's programmers to be really, really clever :-)

Setting the anti-science smear straight

(Yikes! Last post was on Thursday!)

For discussion: Why is it skeptics of climate change (either of the science or the need to do anything) who are always accused of being anti-science, when 90+% of environmentalists, are just as willing to dismiss the economic science that says, "Hey, here's the most efficient way to handle the problem", so that they can go on advocating ridiculously inefficient policies?