In his weekend interview in the Wall Street Journal, my friend and editor Tunku Varadarajan wrote an elegant and gracious account of my views on freedom of speech in the wake of the recent, tragic events in Charlottesville. In this essay, I will elaborate on some of the themes developed there.

When it comes to free speech, the Constitution speaks in broad generalities that start the conversation off in the right direction, but which, standing alone, do not fill in all the missing pieces in a complex puzzle. The relevant text announces that Congress may pass “no law abridging the freedom of speech or the press.” That seemingly strict command is essential to guard against government suppression or censorship of political protests. But the incompleteness of the text raises two difficult questions. First, just what kinds of activities enjoy this constitutional protection? And what justifies limits on that constitutional freedom? Both of these gray areas came into play in Charlottesville, and both will prove more intractable as political strife in the United States deepens. In this dire climate, it is best to return to first principles.

The First Amendment clearly covers the spoken word, written pamphlets, and books. By analogy, it also reaches other expressive activities like drawing, dancing, and acting. But no one could claim that it also protects mayhem, murder, defamation, and deceit. The only way to draw the right line—that between expression and violence—is to recognize that the First Amendment is as much about freedom as it is about speech. The necessary theory of freedom applies equally to all forms of speech and action, and it draws the line at the threat or use of force, even if the former counts as speech and the latter does not. Both must yield to the state’s “police power” to protect public safety and health, even if that phrase is nowhere mentioned in the constitutional text. So the First Amendment offers no protection to people who hold up banks or beat up people, even if they do so to protest against the established order.

The analytical difficulties are compounded, however, given that it is unknown how any given situation will play out. A march could turn violent or remain peaceable, and surely the state can take steps to protect against violence before the march takes place. It can issue permits for parades on public streets and parks, just as it can issue restrictions on who may drive a car or carry a gun. But that permit system must balance between two kinds of error: imposing too weak restrictions on violence or damage to persons or property, or, alternatively, imposing excessive restrictions that prevent the message, however offensive, from being heard.

In an effort to tease out these two strands, standard First Amendment doctrine rightly starts with a presumption in favor of the lawfulness of time, place, and manner restrictions, targeted against noise, nuisances, and of course violence, without regard to the speaker’s viewpoint. But it cannot relax these restrictions for so-called good groups as opposed to bad ones, lest certain political groups be given the power to entrench their views through preferential treatment. Meanwhile, as applied to demonstrations in public parks and public streets, a system of open entry does not work. Some permits are needed to prevent groups from staging protests that turn into armed conflict. But these permits cannot be withheld or granted at will; the state must seek to control violence while allowing speech.

The job is never easy, but it can be done. Last autumn, I saw a well organized and peaceful march against Donald Trump moving south along Central Park West toward the Trump International Hotel. The police erected two sets of barriers, about ten feet apart on each side of the designated parade route, to create a demilitarized zone that would reduce the risks of armed conflict between rival protesters. Similar buffer zones were deployed in the recent Boston rallies to keep rival groups apart.

Nothing of this sort was done, however, to contain the white supremacist and Nazi sympathizers who came to Charlottesville, even though the risk of their violence was far greater. It is a parody of free speech to allow any group, especially one committed to violence, to flaunt weapons, the chief effect of which is intimidation not communication. Any such self-appointed militia must be disarmed before it marches, even if hate groups are allowed to flaunt swastikas in public. It is hard to hold organizers of the march liable for the death and injuries caused by one their followers who plowed into innocent counterdemonstrators with First Amendment rights of their own. All the more reason for strong ex ante protections against potential violence of uncertain origin.

These principles apply to all efforts to use force. Unfortunately, the horrific consequences of Charlottesville have led to proposals to recast First Amendment law in ways that undercut its commitment to viewpoint-neutral protections, as when K-Sue Park, a Critical Race Studies fellow at the U.C.L.A. School of Law, implored the ACLU “to rethink free speech”—that is, to relinquish its commitment to defend the speech rights of groups whose substantive views it finds deeply offensive. To be sure, the ACLU has announced that it will no longer defend groups that conduct protest marches with firearms. The pity is that it did not reach that position before it chose to represent “Unite the Right,” the white supremacy group behind the protest in Charlottesville, which sought a permit to demonstrate near a statue of Robert E. Lee that was slated for removal. The dangers of violence were just too great.

Fortunately, conservatives commentators like the Wall Street Journal’s William McGurn have stoutly defended the ACLU’s decision to defend the rights of all groups, however obnoxious, to engage in peaceful protests. Park rejects that view on the grounds that any set of First Amendment rights should be placed in some larger social context in which the failure of the U.S. to create a “level playing” field on social and political matters is taken into account. In her view, it is the job of the state to overcome the stark inequalities that give wealthy individuals and especially corporations disparate influence over the political culture.

She is of course correct that it is inexcusable to make death threats against black professors who speak out against racism—but she is strangely silent about similar death threats that are directed toward evangelical Christians whose sole offense is to refuse to bake a wedding cake or supply flowers and photography services at same-sex weddings. It is, to say the least, exceedingly dangerous to allow anyone’s view of the overall state of American society to influence their version of First Amendment rights. It would be insane, for example, for the ACLU to defend violence from the alt-left but not the alt-right. Nonetheless, one wonders whether Park would support the United States Civil Rights Commission in its recent refusal to condemn the spate of violence from Antifa.

Regrettably, Park has a wholly distorted view of social inequality because she misses the simple but vital point that inequalities in wealth and position are the necessary consequence of any just society that allows all of its individuals—regardless of race, sex, age, or any political identifier—to obtain the best deal for themselves in a competitive market. Voluntary exchanges benefit all participants, but not to an equal extent. Nevertheless, it is a grievous error to treat these transactions with disdain, suspicion, or hostility. It is of course permissible, and perhaps correct, to tax the rich to help the poor. But it is a non sequitur to imagine that tinkering with a sensible First Amendment system will redress any form of discrimination or inequality, when history shows that progressive policies, especially on economic issues, will typically aggravate the very social inequities they are intended to fix.

Yet, even as the ACLU defends its position on free speech against Park, it concedes far too much to collectivist political institutions. Thus in his letter response to Park, David Cole, the national legal director at the ACLU, correctly warned against allowing government officials to regulate speech “based on who is promoting equality or ‘on the wrong side of history.’” The risk is too great that bad political leaders will abuse their authority in ways that hurt the most vulnerable in society.

But Cole is unwise to insist “that speech rights, like property, privacy and liberty rights, can contribute to inequality,” without pausing to identify the huge social gains that these basic institutions produce. In that one sentence, he exposes the Achilles heel of the modern ACLU, which thinks that it can consistently defend freedom of speech while attacking its sister rights of property and liberty. The illogical result is hounding evangelical Christians under so-called human rights law, while defending their right to denounce the sexual practices of their same-sex customers.

The older view of laissez-faire was far sounder when it held, in the words of The Edinburgh Review of 1843, “that freedom of trade, freedom of thought, freedom of speech, and freedom of action . . . stand or fall together.”