Most libertarians make a moral distinction between violence used to invade someone's rights and violence used in retaliation against someone who has invaded someone's rights. We call the former kind of violence aggression. We regard the latter kind of violence, if it is not excessive, as a legitimate form of self-defense, reparation, or punishment, depending on its purpose. The fundamental law for most libertarians is called the non-aggression principle:
No man or group of men may aggress against the person or property of anyone else.
Libertarians use the word aggression in a special way. For us, aggression is the initiation of the use, or threat, of physical force or violence against the person or property of anyone else.
A whole system of criminal law is packed into the single sentence that expresses the non-aggression principle. To make it clear what libertarians mean by it, let me briefly list the kinds of actions that it prohibits and some of the kinds of actions that it allows.
Crimes against a Person: The non-aggression principle prohibits the initiation of violence against another person. It prohibits murder, rape, battery, kidnapping, imprisonment, enslavement, and torture. The non-aggression principle also prohibits the initiation of threats to use violence against another person, which is called assault. It is not necessary to physically touch someone to commit an assault. For example, it is an assault to point a gun at someone who is not aggressing and then demand something from him.
A more indirect, but still prohibited, kind of assault against someone who is not aggressing is to demand something from him in such a way that it is understood that, if he does not comply, you or your agents will use violence against him. For example, it is an assault when a goon from the Mafia demands a store owner to pay protection money, or when the State enacts a tax law or a victimless-crime law.
Crimes against a Person's Property Rights: The non-aggression principle prohibits the initiation of physical force or violence against another person's property rights. It prohibits destroying, damaging, taking, selling, or using the property of a non-aggressing person without his permission. Destroying or damaging someone's property may be considered violent acts, but it seems a stretch to classify selling, taking, or using someone's property as a violent act, especially when this is done by stealth so that the property owner is unaware of it and is not physically threatened by it. That is why we include "physical force" in our understanding of the non-aggression principle. Selling, taking, and using someone's property are physical acts. They might not involve violence, but they do require some physical force.
Actions That Are Not Crimes: The non-aggression principle permits: (1) all peaceful actions that do not involve the use or the threat of physical force against another person or his property and (2) actions the do involve the use or the threat of physical force or even violence against another person or his property as long as the action is not the initiation of force or the threat of the initiation force against that person or his property.
Actions that fit in the first, peaceful, category include doing things with your own property such as consuming meat, vegetables, drugs, alcohol, or even poison; decorating or mutilating your own body with jewelry, tattoos, scars, make-up, and hair-dos; discriminating against other people based on their race, gender, religion, or any other category whether it is rational or nor; donating your wealth and services to help others, spending it on yourself, or hoarding it; telling the truth, lying, speaking kindly about others, or slandering and libeling them; trading, renting, and making voluntary contracts; having sexual, financial, friendly, or other relationships with consenting adults; and so on.
Actions that fit in the second, more violent, category include using physical force to defend yourself from an attacker or to repossess stolen property. You can also use physical force to defend someone else who is being attacked or to help someone else repossess their property, if they consent to your help.