Networking¶

IP Address Options¶

Bastille includes a number of IP options.

bastille create alcatraz 13.2-RELEASE 192.168.1.50/24 vtnet0

The IP address specified above can be any of the following options.

An IP in your local subnet should be chosen if you create your jail using -V or -B (VNET jail). It is also preferable to add the subnet mask (/24 or whaterver your subnet is) to the IP.
DHCP, SYNCDHCP, or 0.0.0.0 will configure your jail to use DHCP to obtain an address from your router. This should only be used with -V and -B.
Any IP address inside the RFC1918 range if you are not using a VNET jail. Bastille will automatically add this IP to the firewall table to allow outbound access. It you want traffic to be forwarded into the jail, you can use the bastille rdr command.
Any IP in your local subnet without the -V or -B options will add the IP as an alias to the selected interface, which will simply end up sharing the interface. If the IP is in your local subnet, you will not need the bastille rdr command. Traffic will pass in and out just as in a VNET jail.
Setting the IP to inherit will make the jail inherit the entire host network stack.
Setting the IP to ip_hostname will add all the IPs that the hostname resolves to. This is an advanced option and should only be used if you know what you are doing.

Note that jails support specifying an IP without the subnet (/24 or whatever yours is) but we highly recommend setting it, especially on VNET jails. Not doing so can cause issues in some rare cases.

Bastille also supports IPv6. Instead of an IPv4 address, you can specify and IPv6 address when creating a jail to use IPv6. It is also possible to use both by quoting and IPv4 and IPv6 address together as seen in the following example.

bastille create alcatraz 13.2-RELEASE "192.168.1.50/24 2001:19f0:6c01:114c:0:100/64" vtnet0

For the inherit and ip_hostname options, you can also specify -D|--dual to use both IPv4 and IPv6 inside the jail.

Host Network Configuration¶

Bastille will automatically add and remove IP addressess to specified interfaces as jails are started and stopped. Below is an outline of how Bastille handles different types of jail network configs.

VNET mode. For VNET jails (non-bridged) bastille will create a bridge interface and attach your jail to it. It will be called em0bridge or whatever your interface is called. This will be used for the host/jail epairs. Bastille will create/destroy these epairs as the jail is started/stopped.
Bridged VNET mode. For bridged VNET jails, you must manually create a bridge interface to attach your jail to. Bastille will then create and attach the host/jail epairs to this interface when the jail starts, and remove them when it stops.
Alias mode. For classic/standard jails that use an IP that is accessible within your local subnet (alias mode) bastille will add the IP to the specified interface as an alias.
NAT mode. For classic/standard jails that use an IP not reachable in you local subnet, bastille will add the IP to the specified interface as an alias, and additionally add it the the pf firewall table to allow the jail outbound access. If you do not specify an interface, Bastille will assume you have run the bastille setup command and will attemplt to use bastille0 (which is created using the setup command) as it’s interface. If you have not run bastille setup and do not specify an interface, Bastille will error.
Inherit mode. For classic/standard jails that are set to inherit or ip_hostname, bastille will simply set ip4 to inherit inside the jail config. The jail will then function according the jail(8) documentation.
ip_hostname mode. For classic/standard jails that are set to ip_hostname, bastille will simply set ip4 to ip_hostname inside the jail config. The jail will then function according the jail(8) documentation.

Network Scenarios¶

Here’s the scenario. You’ve installed Bastille at home or in the cloud and want to get started putting applications in secure little containers, but how do you get these containers on the network? Bastille tries to be flexible about how to network containerized applications. Four methods are described here.

Home or Small Office
Cloud with IPV4 and multiple IPV6
Cloud with single IPV4 (internal bridge)
Cloud with a single IPV4 (external bridge)

Please choose the option which is most appropriate for your environment.

First a few notes. Bastille tries to verify that the interface name you provide is a valid interface. In FreeBSD network interfaces have different names, but look something like em0, bge0, re0, vtnet0 etc. Running the ifconfig command will tell you the name of your existing interfaces. Bastille also checks for a valid syntax IP4 or IP6 address. When you are testing calling out from your containers, please note that the ping command is disabled within the containers, because raw socket access are a security hole. Instead, install and test with wget/curl/fetch instead.

Shared Interface on Home or Small Office Network¶

If you have just one computer, or a home or small office network, where you are separated from the rest of the internet by a router. So you are free to use private IP addresses.

In this environment, to use Bastille, just create the container, give it a unique private ip address, and attach its ip address to your primary interface.

bastille create alcatraz 13.2-RELEASE 192.168.1.50 em0

You may have to change em0

When the alcatraz container is started it will add 192.168.1.50 as an IP alias to the em0 interface. It will then simply be another member of the hosts network. Other networked systems (firewall permitting) should be able to reach services at that address.

This method is the simplest. All you need to know is the name of your network interface and a free IP on your local network.

Shared Interface on IPV6 network (vultr.com)¶

Some ISP’s, such as Vultr, give you a single ipv4 address, and a large block of ipv6 addresses. You can then assign a unique ipv6 address to each Bastille Container.

On a virtual machine such as vultr.com the virtual interface may be vtnet0. So we issue the command:

bastille create alcatraz 13.2-RELEASE 2001:19f0:6c01:114c::100 vtnet0

We could also write the ipv6 address as 2001:19f0:6c01:114c:0:100

The tricky part are the ipv6 addresses. IPV6 is a string of 8 4 digit hexadecimal characters. At vultr they said:

Your server was assigned the following six section subnet:

2001:19f0:6c01:114c:: / 64

The vultr ipv6 subnet calculator is helpful in making sense of that ipv6 address.

We could have also written that IPV6 address as 2001:19f0:6c01:114c:0:0

Where the /64 basicaly means that the first 64 bits of the address (4x4 character hexadecimal) values define the network, and the remaining characters, we can assign as we want to the Bastille Container. In the actual bastille create command given above, it was defined to be 100. But we also have to tell the host operating system that we are now using this address. This is done on freebsd with the following command

ifconfig_vtnet0_alias0="inet6 2001:19f0:6c01:114c::100 prefixlen 64"

At that point your container can talk to the world, and the world can ping your container. Of course when you reboot the machine, that command will be forgotten. To make it permanent, prefix the same command with sysrc

Just remember you cannot ping out from the container. Instead, install and use wget/curl/fetch to test the connectivity.

Virtual Network (VNET)¶

(Added in 0.6.x) VNET is supported on FreeBSD 12+ only.

Virtual Network (VNET) creates a private network interface for a container. This includes a unique hardware address. This is required for VPN, DHCP, and similar containers.

To create a VNET based container use the -V option, an IP/netmask and external interface.

bastille create -V azkaban 13.2-RELEASE 192.168.1.50/24 em0

Bastille will automagically create the bridge interface and connect / disconnect containers as they are started and stopped. A new interface will be created on the host matching the pattern interface0bridge. In the example here, em0bridge.

The em0 interface will be attached to the bridge along with the unique container interfaces as they are started and stopped. These interface names match the pattern eXb_bastilleX. Internally to the containers these interfaces are presented as vnet0.

If you do not specify a subnet mask, you might have issues with jail to jail networking, especially VLAN to VLAN. We recommend always adding a subnet to VNET jail IPs when creating them to avoid these issues.

VNET also requires a custom devfs ruleset. Create the file as needed on the host system:

## /etc/devfs.rules (NOT .conf)

[bastille_vnet=13]
add include $devfsrules_hide_all
add include $devfsrules_unhide_basic
add include $devfsrules_unhide_login
add include $devfsrules_jail
add include $devfsrules_jail_vnet
add path 'bpf*' unhide

Lastly, you may want to consider these three sysctl values:

net.link.bridge.pfil_bridge=0
net.link.bridge.pfil_onlyip=0
net.link.bridge.pfil_member=0

Below is the definition of what these three parameters are used for and mean:

net.link.bridge.pfil_onlyip Controls the handling of non-IP packets
which are not passed to pfil(9). Set to 1 to only allow IP packets to pass (subject to firewall rules), set to 0 to uncondi- tionally pass all non-IP Ethernet frames.

net.link.bridge.pfil_member Set to 1 to enable filtering on the incom-
ing and outgoing member interfaces, set to 0 to disable it.

net.link.bridge.pfil_bridge Set to 1 to enable filtering on the bridge
interface, set to 0 to disable it.

Bridged Network (VNET bridged)¶

To use a bridged VNET setup the first thing you have to do is to create a bridge interface on your system. This is done with the ifconfig command and will create a bridged interface named bridge0:

Then you need to add your system’s network interface to the bridge and bring it up (substitute your interface for em0).

Optionally you can rename the interface if you wish to make it obvious that it is for bastille:

To create a bridged container you use the -B option, an IP or DHCP, and the bridge interface.

All the epairs and networking other than the manually created bridge will be created for you automagically. Now if you want this to persist after a reboot then you need to add some lines to your /etc/rc.conf file. Add the following lines, again, obviously change em0 to whatever your network interface on your system is.

Regarding Routes¶

Bastille will attempt to auto-detect the default route from the host system and assign it to the VNET container. This auto-detection may not always be accurate for your needs for the particular container. In this case you’ll need to add a default route manually or define the preferred default route in the bastille.conf.

bastille sysrc TARGET defaultrouter=aa.bb.cc.dd
bastille service TARGET routing restart

To define a default route / gateway for all VNET containers define the value in bastille.conf:

bastille_network_gateway=aa.bb.cc.dd

This config change will apply the defined gateway to any new containers. Existing containers will need to be manually updated.

Virtual Network (VNET) on External Bridge¶

To create a VNET based container and attach it to an external, already existing bridge, use the -B option, an IP/netmask and external bridge.

bastille create -B azkaban 13.2-RELEASE 192.168.1.50/24 bridge0

Bastille will automagically create the interface, attach it to the specified bridge and connect / disconnect containers as they are started and stopped. The bridge needs to be created/enabled before creating and starting the jail.

Public Network¶

In this section we describe how to network containers in a public network such as a cloud hosting provider who only provides you with a single ip address. (AWS, Digital Ocean, etc) (The exception is vultr.com, which does provide you with lots of IPV6 addresses and does a great job supporting FreeBSD!)

So if you only have a single IP address and if you want to create multiple containers and assign them all unique IP addresses, you’ll need to create a new network.

loopback (bastille0)¶

What we recommend is creating a cloned loopback interface (bastille0) and assigning all the containers private (rfc1918) addresses on that interface. The setup I develop on and use Bastille day-to-day uses the 10.0.0.0/8 address range. I have the ability to use whatever address I want within that range because I’ve created my own private network. The host system then acts as the firewall, permitting and denying traffic as needed.

I find this setup the most flexible across all types of networks. It can be used in public and private networks just the same and it allows me to keep containers off the network until I allow access.

Having said all that here are instructions I used to configure the network with a private loopback interface and system firewall. The system firewall NATs traffic out of containers and can selectively redirect traffic into containers based on connection ports (ie; 80, 443, etc.)

First, create the loopback interface:

ishmael ~ # sysrc cloned_interfaces+=lo1
ishmael ~ # sysrc ifconfig_lo1_name="bastille0"
ishmael ~ # service netif cloneup

Second, enable the firewall:

ishmael ~ # sysrc pf_enable="YES"

Create the firewall rules:

/etc/pf.conf¶

ext_if="vtnet0"

set block-policy return
scrub in on $ext_if all fragment reassemble
set skip on lo

table <jails> persist
nat on $ext_if from <jails> to any -> ($ext_if:0)
rdr-anchor "rdr/*"

block in all
pass out quick keep state
antispoof for $ext_if inet
pass in inet proto tcp from any to any port ssh flags S/SA modulate state

Make sure to change the ext_if variable to match your host system

interface. - Make sure to include the last line (port ssh) or you’ll end up locked out.

Note: if you have an existing firewall, the key lines for in/out traffic to containers are:

nat on $ext_if from <jails> to any -> ($ext_if:0)

The nat routes traffic from the loopback interface to the external interface for outbound access.

rdr-anchor "rdr/*"

The rdr-anchor "rdr/*" enables dynamic rdr rules to be setup using the bastille rdr command at runtime - eg.

bastille rdr TARGET tcp 2001 22 # Redirects tcp port 2001 on host to 22 on jail
bastille rdr TARGET udp 2053 53 # Same for udp
bastille rdr TARGET list        # List dynamic rdr rules
bastille rdr TARGET clear       # Clear dynamic rdr rules

Note that if you are redirecting ports where the host is also listening (eg. ssh) you should make sure that the host service is not listening on the cloned interface - eg. for ssh set sshd_flags in rc.conf

sshd_flags="-o ListenAddress=<host-address>"

Finally, start up the firewall:

ishmael ~ # service pf restart

At this point you’ll likely be disconnected from the host. Reconnect the ssh session and continue.

This step only needs to be done once in order to prepare the host.

local_unbound¶

If you are running “local_unbound” on your server, you will probably have issues with DNS resolution.

To resolve this, add the following configuration to local_unbound:

server:
interface: 0.0.0.0
access-control: 192.168.0.0/16 allow
access-control: 10.17.90.0/24 allow

Also, change the nameserver to the servers IP instead of 127.0.0.1 inside /etc/rc.conf

Adjust the above “access-control” strings to fit your network.