In biology-adjacent circles around the world, this week began with discussion of the CASP14 results, where the second version of AlphaFold achieved outstanding results in protein structure prediction. The title of DeepMind's press release alone—“AlphaFold: a solution to a 50-year-old grand challenge in biology”—split the community so thoroughly that the battles on social media were still raging several days later. This is probably a good reason to put a few thoughts on the subject into words. Without any claim to objectivity, of course.
What happened?
Every two years, a competition called the Critical Assessment of Techniques for Protein Structure Prediction, better known as CASP, takes place. Participants are given protein sequences whose structures have already been determined but not yet published. Using only those sequences, they must predict the structures of the proteins. Prediction tasks are divided into categories by difficulty—depending on how dissimilar a protein is from all known proteins—and by solution method, from fully automated to varying degrees of human involvement.
Traditionally, the most difficult category is considered to be ab initio modelling: predicting a structure “from scratch”, without additional information about similar proteins. For a long time, there were virtually no breakthroughs in this area. Then, in 2018, a team called A7D appeared, beating every other team by dozens of arbitrary units—we will discuss those later. The name concealed AlphaFold, a new algorithm from DeepMind, by then already famous for its victories on the Go board. Naturally, the solution used deep neural networks, causing an enormous stir among people involved in the field and, especially, among the general public.
Over the next two years, the algorithm was published and its code released, but no revolution occurred in structural biology. This year, the result improved dramatically: the new version of AlphaFold's lead over second place was more than 2.5-fold—in the same arbitrary units—creating an incredible wave of hype on one side and outrage on the other. Before criticising the views of pessimists and optimists, let us examine the problem statement and the interpretation of the results.
The problem
First, we need to settle the terminology. In its press release, DeepMind uses the phrase “protein folding problem”. Formally, protein folding is the process, extended over time, by which a chain of amino acids becomes a spatial structure that performs some biological function. The dynamics and kinetics of that process—put simply, though not entirely accurately, its trajectory and speed—determine both the final structure and, for example, the efficiency with which it can be produced. The latter is of particular interest to anyone who manufactures proteins, including pharmaceutical companies.
What teams actually do at CASP has a different name: protein structure prediction. Given an amino-acid sequence, they predict only the final structure, without accounting for the intermediate process. This is more than enough for an enormous number of practical tasks—one could cautiously say most of them. Moreover, at our present level of development, this is the only version of the problem that is even somewhat tractable, and then only with major caveats, whereas our progress towards modelling the folding process itself is extremely slow. Simulations of very short proteins—the video below shows 39 amino acids, tens of times fewer than the proteins we usually work with—may take several days on distributed GPU clusters. Plausible trajectories for proteins of genuine interest, containing hundreds of amino acids, remain entirely out of reach.
As a result, most people both inside and outside the field use “folding” to mean structure prediction. DeepMind blurs the concepts slightly by declaring a solution to a “50-year-old grand challenge” in the title, but immediately clarifies that it means protein structure prediction. This imprecision nevertheless gave purists room to grumble that the work had nothing to do with the folding problem. We will return to this among the other objections.
Even the reduced problem of protein structure prediction—reduced relative to folding itself—is still extremely difficult because it requires determining the position of every atom in every amino acid. Real proteins can involve thousands or tens of thousands of three-dimensional coordinates, all seeking to minimise an energy potential. We have a reasonable idea of what a function approximating that potential energy should look like. But the dimensionality of the problem completely rules out global optimisation approaches, while local optimisation is unsuitable because the potential is, to put it mildly, non-convex. If the last few sentences were unclear, the key point is this: although we know how to find the solution by brute force, no computer can do so in any reasonable amount of time. We therefore have to resort to various tricks. We will examine them using a tiny protein of 28 amino acids as our example.
The most common approach to structure prediction is template-based prediction, also known as homology modelling. It is founded on the idea that similar amino-acid sequences produce similar three-dimensional protein structures. Given a sufficiently large database of high-quality experimental structures, we can replace a number of fragments to obtain a reasonably good all-atom model. Additional knowledge about a particular class of proteins makes it possible to fill in differing or unknown fragments especially well. Homology modelling therefore reflects the idea that if we cannot solve the problem for all proteins, we can at least solve it for each protein. It is not universal, but it works.
Problems obviously begin when the structure database contains no protein with a closely related sequence, which happens quite often. This task is considered so difficult that it is reduced even further. It keeps the same name, of course. This is biology.
How can we reduce the problem? We can discard most of the atoms, for example. First—with a catch already—we will remove the side-chain atoms, making every amino acid anonymous and retaining only the backbone atoms. Such a structure is no longer suitable for many tasks, but it still gives us an excellent understanding of the protein's overall organisation. And, looking ahead, we are reasonably good at reconstructing side-chain positions once we have the backbone. This is what we obtain:
We can go further and represent each amino acid by a single atom—the carbon from which the amino-acid side chain “grows”. This is an extremely coarse model, but it greatly reduces the number of atoms, making it equal to the number of amino acids. Such a structure is practically useless for solving applied problems, of course. But perhaps we can extend it later. This is precisely what the ab initio modelling category of CASP proposes.
Arbitrary units
We have fed our amino-acid sequence into the predictor and obtained a set of coordinates for our atoms. How do we determine whether the result is any good? We need to align the molecules, positioning them so that corresponding atoms are as close as possible, and then calculate the distances between corresponding pairs of atoms. One such distance between atoms in two models of the same structure is shown below:
Once we have found the distance between every pair of atoms, we can express the similarity of the structures with a single number: RMSD, the root-mean-square deviation of the atoms from one another. The RMSD between a model and the “true”—that is, experimentally determined—structure is our measure of model quality.
The obvious problem with RMSD is averaging. A prediction in which every atom but one is correct and the remaining atom is very far away—as in the image above—can receive the same score as a prediction in which every atom is somewhat wrong, even though the two have very different practical value. CASP therefore uses a score based on the Global Distance Test, or GDT. GDT measures the proportion of model atoms that, after alignment, lie no farther than a given cut-off from their corresponding atoms in the “true” structure. GDT therefore ranges from 0 to 1 and is usually expressed as a percentage. CASP uses cut-offs of 1, 2, 4, and 8 Å, averages the resulting four values, and reports a single “total score”: GDT_TS.
In fact, the CASP organisers go even further, torturing even the resulting GDT_TS values by converting them into z-scores across all predictions for a given structure, removing outliers, and clipping excessively low values at either -2 or 0. The z-scores for every structure predicted by a team are then summed and averaged. This leaves us with four measures: total z-scores at cut-offs of -2 and 0, and mean z-scores at cut-offs of -2 and 0. This is usually done to avoid having to declare a winner, allowing several teams at once to say that they placed number 1, 2, 3, or whatever in CASP by a particular measure.
This time, AlphaFold2 won unconditionally by all four measures.
In its press release, DeepMind also reports median GDT_TS values of 92.4 across all targets and 87.0 for particularly difficult proteins. By its estimate, the first corresponds to an RMSD of 1.6 Å, formally comparable to the quality of structures obtained by X-ray crystallography. This claim, like many others, is somewhat speculative.
How does this thing work?
We do not know for certain how AlphaFold2 works, but we can make inferences from the press release, the published paper, and the code for the first version of the algorithm. At this stage, that may even be a good thing, because I do not have to agonise over how much implementation detail to include.
Instead, let us examine the main ideas that made the whole endeavour succeed. In my opinion, the decisive factor was not an ingenious neural-network architecture. Those architectures will of course grow ever more complex and squeeze out ever more arbitrary units in competitions. But the main key to success was the use of convolutional neural networks together with a useful observation—not made by DeepMind—that enabled the data to be prepared intelligently and an appropriate output format to be chosen. We will begin with the latter. But first, to be safe, let me repeat: we are examining a couple of general ideas. I am deliberately omitting some details specific to AlphaFold.
AlphaFold does not predict a protein structure.
Let us pause for a moment and consider how a protein structure can be expressed mathematically. Until now, we have continually assumed without saying so that a structure means atom coordinates—whether for every atom, the backbone atoms, atoms, or something else. In practice, however, this is a very poor representation because it is not unique. We normally treat a prediction as some deterministic function: it takes a sequence as input and always returns one single answer. But which member of the infinite set of coordinate representations is “canonical”?
It is also completely unclear how machine-learning models could be trained on such data. They need to discover patterns, but if every protein has a different rotation and translation in space, finding anything in entirely meaningless coordinates is hopeless from the start.
DeepMind was not the first to realise this, of course, and the search for an invariant—a representation of a structure that does not depend on its rotation or translation in space—began a long time ago. For decades, the classic solution was based on angles.
Let us conduct a simple experiment on the same small structure we have used throughout. We will measure the distances between every pair of consecutive atoms.
Suddenly, we find that the distances are nearly identical. Moreover, if we take a different protein and examine the distances in that structure, we see exactly the same values. I went to the trouble of doing this for several consecutive atoms in the NTL9 protein structure from the video near the beginning of the post:
This significant observation suggests that we can construct our structure using only the direction of each next atom relative to the previous one, while treating the distances as constant.
If we worked in coordinate space, such a direction would be given by a unit vector. But we remember that we dislike coordinate space because it is absolute, while our proteins can move through it arbitrarily. It would be useful to specify the direction relatively as well, without reference to global space. For that, we need angles.
The image above should convincingly demonstrate that a vector's direction in three-dimensional space can easily be specified by two angles—we can think of them as latitude and longitude, as on a globe. To define those angles, however, we need a plane relative to which they are measured and a point on that plane from which the direction is specified.
The point is straightforward: it is simply the previous atom. A plane, as we know, can be defined by three points. We can take the three previous atoms, and the two angles then arise naturally. The first is the plane angle between three atoms—the current atom and the previous two. The second is the dihedral angle across all four points.
This representation is quite convenient: for amino acids, we predict dihedral angles and plane angles, which fully determine the structure. It has only one problem: it does not work.
More precisely, no one has managed to make any model predict angles from sequence with decent accuracy. Many such algorithms exist, of course, and all perform with varying degrees of awfulness. Why this happens is probably a subject for another post, but briefly, it seems to be because angle prediction solves a very local structural problem and takes no account of interactions between amino acids that are far apart.
This is where another, much simpler invariant enters the picture. Unlike angles, it can be explained in a single sentence: an invariant representation of a structure is the matrix—or table—of all pairwise distances between its atoms. This representation is independent of rotations or translations of the structure in three-dimensional space and contains information about interactions—the proximity of amino acids—regardless of their separation in the sequence. Humanity has also known for a very long time how to recover a set of coordinates efficiently from a distance matrix. That too is a topic for another post, but one of the oldest methods can be explored here.
0.0 3.83271 6.01989 6.58519 … 25.678 24.6388 24.0295
3.83271 0.0 3.83997 6.59813 23.5736 22.5554 21.486
6.01989 3.83997 0.0 3.82077 20.5022 19.164 18.2255
6.58519 6.59813 3.82077 0.0 20.6506 19.0817 18.7819
10.0775 9.08123 5.37992 3.8444 17.0151 15.2753 15.0675
11.085 10.8644 7.81588 5.66081 … 16.316 15.1335 15.7933
14.4674 13.5387 10.1526 8.7795 12.6211 11.3417 12.3157
16.4536 15.7524 12.8618 11.6161 11.6552 11.4539 13.233
13.8395 13.8638 11.4989 9.82352 15.386 15.1816 16.6443
10.8774 10.5668 8.77125 8.06832 16.6945 16.5649 17.2367
7.27608 6.78375 5.36449 5.38269 … 18.9654 18.3924 18.3666
8.49109 6.13378 4.79525 7.04693 17.8971 17.399 16.6753
8.15412 4.97326 5.92916 9.24726 20.7562 20.3603 19.1268
⋮ ⋱ ⋮
14.6443 11.3347 9.85678 12.5927 14.7822 14.6226 12.817
13.4068 10.4863 7.84574 9.82696 13.6534 12.7312 11.2792
16.7364 13.568 10.9384 12.9258 12.5891 11.3365 8.88843
18.6057 15.4437 13.2634 15.3405 11.0792 11.0058 8.80482
17.6728 15.0817 12.5043 13.7692 … 9.06338 9.17429 8.31031
19.7891 17.344 14.2232 14.8957 6.80036 5.72403 4.80765
22.4295 19.6197 16.8814 18.1163 6.56397 6.69385 4.6497
22.6069 20.0842 17.5949 18.6529 5.71467 7.68905 7.47085
22.0187 19.9578 17.0287 17.2996 3.8602 5.29157 6.763
25.678 23.5736 20.5022 20.6506 … 0.0 3.81372 6.17142
24.6388 22.5554 19.164 19.0817 3.81372 0.0 3.84365
24.0295 21.486 18.2255 18.7819 6.17142 3.84365 0.0
The table of pairwise distances between the atoms of our model protein
This is the matrix AlphaFold seeks. Coordinates are then recovered using existing tools and gradient descent. AlphaFold also predicts distances not only between atoms but also between atoms, the first atoms in amino-acid side chains.
Now that we understand what AlphaFold predicts and how that prediction is turned into a structure, let us make one more bold statement.
The input to AlphaFold is not a protein's amino-acid sequence.
Incidentally, this is the main difficulty in using the code released by DeepMind, although there is a community implementation that remedies the situation. To obtain the structure of a sequence of interest, the sequence must be prepared correctly. And that preparation contains the second major ingredient of the method's success.
A sequence by itself obviously carries no information about interactions between amino acids. No matter how ingenious and complex the neural network, extracting that information is difficult. We can simplify the task by telling the algorithm in advance that certain positions in the sequence may be related to one another. But where can we obtain that information?
The absence of structural information about our protein does not mean that we know nothing about it. Since the invention of high-throughput sequencing methods, people have collected and annotated a truly colossal number of gene sequences encoding different proteins. Among them, there are bound to be sequences at least partly similar to the one that interests us.
Most likely, these sequences are similar for a reason: the proteins they encode perform related functions, whether in the same organism or another one. This obviously implies that the structures of those proteins should also be similar.
“So what?” you might ask. We do not know the structures of those sequences. Often, we do not even know their names. How can it help us that someone observed a similar sequence in some organism? Quite a lot, as it turns out.
Let us return to our protein and focus on a pair of interacting amino acids. Imagine that we replace one of them. The interaction will probably be lost, the protein will change its structure, and it may even cease to perform its function. The organism in which the mutation occurs will most likely suffer and be less likely to reproduce.
On the other hand, if the amino acid is replaced by one capable of forming the same or a similar interaction, the structure will change little and the function will probably be preserved.
This observation bears directly on our sequences. We are looking at sequencing results from viable organisms, so we infer that their genes probably work as intended. Similar sequences will therefore have similar structural interactions. Positions responsible for those interactions are likely to be correlated: they change together. Calculating such correlations across a collection of sequences similar to ours is straightforward.
Alongside the sequence, which carries very little information by itself, we thus obtain a square matrix describing the correlation of every pair of positions, usually called a coevolution matrix. Incidentally, it has the same dimensions as the algorithm's answer—the pairwise distance matrix. Both are for a sequence of length . This is convenient because it greatly simplifies the construction of the convolutional neural network that will process the data.
The AlphaFold2 press release briefly mentions that the new neural network was trained end to end. We can cautiously infer that the network now handles every stage from identifying coevolution to producing the final structural representation. On the other hand, the accompanying image shows a separate search for related sequences—“Genetics search & embed”—which apparently still uses more classical methods. Either way, it is too early to judge from a PR announcement alone. We will wait for the paper.
If you have read this far, you have already realised that I deceived you: under the guise of commentary on religious disputes on social media, I gave you an article about the protein structure prediction problem and the ideas that make it tractable. To be honest, tracing the evolutionary process that brought us to the CASP14 results is genuinely more interesting to me than discussing individual opinions about the applicability of those results. But let us nevertheless devote a few rapid-fire paragraphs to them.
What the pessimists say
But this is not folding!
Several times, I have encountered claims that DeepMind is deceiving everyone by substituting one problem for another. It solved protein structure prediction, not the folding process; therefore, such people say, the result has no value.
I have already noted that “folding problem” is routinely used to refer to structure prediction, so I place these critics among the purists—which is not necessarily a bad thing. In my view, however, the remarks are not particularly important, at least while we believe that an amino-acid sequence is sufficient to predict a structure. For the most part, we do believe this, though there are nuances to which we will return shortly.
The claim that structure prediction has little practical value strikes me as a far more harmful distortion. Knowing the path by which a protein folds into its final form would indeed be extremely useful for solving several important problems, but the overwhelming majority of practical tasks require high-quality structures—though they are not exhausted by structures alone. The possibility of no longer spending two years obtaining such structures experimentally would be a genuine breakthrough.
It works only for single-domain proteins
Many proteins do not form one compact domain but consist of several independently packed segments called domains. A typical example, and one close to my heart, is an antibody.
This is a serious limitation, but not an insurmountable one. I see no fundamental obstacle to obtaining the structures of multi-domain proteins. The current solution, however, does not yet seem capable of handling them.
Even more interesting will be methods that can recover the structures of proteins composed of several independent amino-acid chains. Research on antibodies, once again, suggests that this poses no special problem either and that a neural network can be modified to accept several chains at once. Moreover, assembling such proteins is closer to finding how molecules fit together—molecular docking—and may formally fall outside structure prediction altogether. But that is a matter of taste.
It predicts averaged structures
The idea that an amino-acid sequence uniquely determines a structure is actually completely false. The conditions in which a protein folds—the concentrations of various ions, the presence of other proteins nearby, and so on—have no less influence. The resulting structures may differ entirely under different conditions.
AlphaFold, like every other machine-learning algorithm for this task, is trained on experimental structures from the Protein Data Bank, or PDB, which contains proteins under a wide variety of conditions. It is easy to find numerous proteins there with identical sequences but different structures.
For us, this means that AlphaFold averages structures, producing results that may correspond to no real conditions and are therefore useless.
In practice, of course, the situation is more complicated. There are indeed whole classes of proteins that make sense only in the context of their conditions, because their mechanism of action consists precisely of behaving differently in different situations. But these objects are clearly not the majority.
In drug development, we deal predominantly with objects for which conditions have no major effect. Antibodies targeting surface receptors, molecules targeting kinases, and viral capsids for gene therapy are generally designed without accounting for conditions, precisely for the averaged case. When conditions do matter, we turn to molecular simulations, where we can “salt” our protein appropriately. Structure prediction is not the only method in structural bioinformatics; the other methods simply have to be used intelligently.
We must honestly acknowledge, however, that some problems cannot be solved this way. Structural changes under different conditions may be so great that computational methods are not yet capable of simulating their dynamics. In those cases, we need structures obtained under the conditions of interest. The problem is that if we divide the training set into subsets for particular conditions, the 170,000 structures in the PDB may no longer be enough to train a model properly. Or perhaps they will. Either way, AlphaFold cannot currently do this.
We do not know the method's domain of applicability
When we predict the structures of unknown proteins from new families, we have no idea what the answer should look like and therefore cannot verify the prediction's truth. We simply receive an answer without knowing when it is right and when it is wrong.
This situation is hardly unusual in deep learning; normally we know even less. Can you interpret and understand why GPT-3, with its millions of parameters, produced one answer rather than another? Not a chance. That does not stop people from using it. We use equally uninterpretable models in computer vision, stock trading, self-driving cars, and many other fields. Usually, then, this is not considered a problem.
The structural bioinformatics community—and people working in biotechnology more generally—is more scrupulous about this and dislikes using results whose plausibility cannot be assessed. That is entirely right. But I fear that blind trust in machine learning will one day engulf this refuge too.
The accuracy is still insufficient for rational design
Supporters of this view cite cases where moving atoms by 1 Å causes a substantial change in protein function. I want to ask them a question: how many experimentally determined structures have you seen at a resolution better than 1 Å? To spare everyone the trouble, I will answer it myself. There are 803. Out of 170,000.
DeepMind's claimed accuracy is about 1.6 Å, which, judging by the chart, is even better than most experimental structures. That might be the end of the commentary on this point, but there is one subtlety: the accuracy estimate applies only to atoms, while the chart above refers to all atoms in a structure.
What does that mean? Honestly, I do not know. Humanity has achieved some success in reconstructing a complete protein structure from the backbone alone, through both optimisation and, once again, machine learning. But no one has yet combined AlphaFold with these methods and measured the resulting accuracy. We therefore have no idea what the real accuracy of AlphaFold—or, more precisely, of methods that use it—is when producing structures suitable for practical work.
What the optimists say
The optimists are much less interesting to discuss because many of them have nothing to do with structural biology—or biology of any kind—so their remarks tend to be emotional rather than substantive. I selected a few that seemed important.
A 2.5-fold lead over second place!
I think we have examined the arbitrary units used to compare algorithms in sufficient detail. Personally, the only thing I learned was that I cannot draw any objective conclusions from these comparisons.
The figure from the same press release that compares results across years strikes me as far more informative. It too shows substantial progress, even though the tasks become harder every year.
The progress really is significant. But I would like to see the GDT_TS values for the other solutions, to understand whether they are perhaps not as far behind as the manipulated z-score results make them appear.
Sometimes even a rough structure is enough
It is hard to disagree that a rough structure is better than no structure. In practice, however, I do not know a single important problem that can be solved with a -only structure. Perhaps it could help model protein behaviour in a coarse-grained representation, when one needs to simulate a long period and observe the protein's behaviour. But my own experience with such models has been extremely negative.
Since my limited horizons leave me with nothing else to say on the subject, I will add the observation already made above: we do know how to turn incomplete structures into complete ones. As noted earlier, however, it is unclear how well the combination will work.
The underlying method is what matters; it has many applications
Yes, yes, and yes again! One could say that this is exactly why I am an optimist. Setting aside the practical value of AlphaFold in its present form, it seems difficult to deny that we now possess an interesting new method with potentially broad applicability to related tasks.
The link between coevolution—correlation between sequence positions—and distances between atoms is a powerful tool well suited to machine learning. In support of this claim, the two years since the first AlphaFold results were released have produced several interesting solutions to specific applied problems.
An important example for me is Jeffrey Gray's laboratory, which developed an excellent method for predicting antibody loops. This is a highly relevant applied problem for modern pharmaceuticals and one that is still solved very poorly. Gray's team, incidentally, abandoned the coevolution matrix—which is understandable, since there are far fewer antibody sequences than protein sequences in general, and their variability is much higher—but introduced a couple of other tricks specific to antibodies. And the method took off.
In my view, this is exactly what gives us hope that methods for important industry problems will appear soon. As someone who works in the industry, that is what I need—not a single algorithm for a fundamental problem whose output I cannot use in practice.
Conclusion, or what I say
Protein structure prediction is undoubtedly an important tool for analysing biological systems and for rational biological design. But we must clearly understand that it is not a silver bullet; it is one tool among many. Even a perfect solution to this problem would not resolve every challenge in structural biology or make drug development many times faster or cheaper. The appearance of such a solution would nevertheless move the industry forward substantially.
Unfortunately, we do not yet have that solution. Even an excellent prediction is not yet a structure prediction. AlphaFold's value lies not in its practical applicability, but in the methodology behind it, which can be used to solve many related problems. It is therefore a major breakthrough, and in my view the entire field owes DeepMind a great deal of gratitude.
There is an enormous amount of work ahead on many interesting problems. Computational bioinformatics is becoming more difficult and more fascinating, but its practitioners also have more opportunities than ever. We may not solve every problem in one stroke, but we will solve every one of them eventually!