Handle NetworkExtension: 2. analysis of the official Demo source code NEPacketTunnelProvider use part

In 2015, Apple’s official release of NetworkExtension API and official DEMO at WWDC 2015 was very exciting, when many developers developed their own NE related programs based on official DEMO

The official demo is here

After so many years, however, the swift version has also evolved to version 3.1, and for anyone like me who learns the syntax of swift 3 directly, it is also wonderful that demo can be updated to 3

Handle NetworkExtension: 2. analysis of the official Demo source code NEPacketTunnelProvider use part
update history

Thanks Apple daddy and program apes, and also upgraded DEMO to swift 3., so let’s download it to compile and see


A compiler, more than 100 error, mostly grammar problems, not sure whether to retain the syntax of 2.x or 3, in short, in the current 3.1 edition of swift, crazy error……

This amendment in debating whether or not to error (estimated workload is not small), fortunately, the amendment version of —&gt was found in GitHub; GitHub address
code to download, compile, run, and the end of the!

Handle NetworkExtension: 2. analysis of the official Demo source code NEPacketTunnelProvider use part
, huh?

Well, no nonsense, or talk about the principle, and the purpose of this article: we learned how to use the NEPacketTunnelProvider, and the principle of communication.

First of all, tell me what NEPacketTunnelProvider is for

What does 1. NEPacketTunnelProvider do?

That’s what the officials say

Handle NetworkExtension: 2. analysis of the official Demo source code NEPacketTunnelProvider use part
official statement

In short, the system is sending network requests to go through these goods packaging to encrypted transmission we connect to the server. The channel is a flow exit gate. How to package, how to encrypt in advance secretly by an unknown path, here.

That is to say, we can set up a TCP request in this connection and connect it to our receiving server to make a channel

2., we analyze DEMO SimpleTunnel related source code

2.1 establish TCP connection

The previous article said that the build request occurred in the startTunnel method. The source code for Demo is as follows

Begin the process of establishing / / / the tunnel. override func startTunnel (options: [String NSObject], completionHandler: @escaping? (Error?) -> Void) {let newTunnel = ClientTunnel (newTunnel.delegate) = self if = newTunnel.startTunnel let error (self) {completionHandler (error as NSError)} else {/ / Save the completion handler for when the tunnel is fully established. pendingStartCompletion = completionHandler tunnel = newTunnel}}

Here the ClientTunnel is in fact a client channel class definition in SimpleTunnelServices.
first create a channel, and then launched a channel request. The core content is newTunnel.startTunnel (self).

Well, let’s go in and see what’s going on inside

Start the TCP connection to / / / the tunnel server. open func startTunnel (provider: NETunnelProvider _) -> SimpleTunnelError? Let serverAddress provider.protocolConfiguration.serverAddress else {guard = {return.BadConfiguration} let endpoint: NWEndpoint if let colonRange serverAddress.rangeOfCharacter (from: = CharacterSet ("charactersIn:"), options: [], range: nil The server is specified) {/ / in the configuration as < host> < port> let: hostname = serverAddress.substring (with: serverAddress.startIndex..< colonRange.lowerBound let portString serverAddress.substring (with:) = serverAddress.index (after: colonRange.lowerBound)..< serverAddress.endIndex; guard HO)! Stname.isEmpty & & portString.isEmpty; else return.BadConfiguration}! {endpoint = NWHostEndpoint (hostname:hostname, port:portString)} else {/ / The server is specified in the configuration as a Bonjour service name. endpoint NWBonjourServiceEndpoint (name: = serverAddress, type:Tunnel.serviceType, domain:Tunnel.serviceDomain) off the connection} / / Kick to the server. connection = provider.createTCPConnection (to: endpoint, enableTLS:false, tlsParameters:nil, delegate:nil) for notificationes when the / / Register connection status changes. connection!.addObserver (self, forKeyPath: "state", options:.Initial, context: & connection return nil})

Look at the content should be able to see roughly, first judge whether the external address is normal ip:port format, if not, then use the Bonjour service to generate the default address. No eggs

If it is normal, then we use the IP port and a new Endpoint, is the target endpoint server. After a new connection to the target server to connect TCP. Provider.createTCPConnection (to: endpoint, enableTLS:false, tlsParameters:nil, delegate:nil)
after adding the observation to the connection state change

2.2 let’s look at what happened after the connection was built

Switch connection case.Connected: if let!.state {remoteAddress = self.connection!.remoteAddress as? NWHostEndpoint = remoteAddress.hostname {remoteHost} / / Start reading messages from the tunnel connection. (readNextPacket) the delegate know that / / Let the tunnel is open delegate?.tunnelDidOpen (self) case.Disconnected: closeTunnelWithError (connection.Error as NSError!?) case.Cancelled: connection!.removeObserver (self, forKeyPath: state, context:& connection) connection = nil delegate?.tunnelDidClose (self) default: break}

After establishing a connection, read the data returned by the server. The server is in order to get the distribution of visual information such as IP address. The other two is simple.
so we look at this section of the key (readNextPacket) what have you done

Read a SimpleTunnel packet from / / / the tunnel connection. func readNextPacket (guard) {let targetConnection = connection else closeTunnelWithError (SimpleTunnelError.badConnection as NSError) {return} / / First, read the total length of the packet. targetConnection.readMinimumLength (MemoryLayout< UInt32>.Size maximumLength: MemoryLayout< UInt32>.Size) {data, error in if let readError = error {simpleTunnelLog ("Got an error on the tunnel connection: (readError) / self.closeTunnelWithError (readError") as NSError? Return let lengthData}) = Data Guard lengthData!.count MemoryLayout< UInt32>.Size = else {simple; TunnelLog ("Length data (length / (lengthData!.count))! = sizeof ((UInt32) / (MemoryLayout< UInt32>.Size) self.closeTunnelWithError (SimpleTunnelError.internalError") as NSError return var totalLength: UInt32}) = 0 (lengthData as! NSData.GetBytes (&); length: MemoryLayout< totalLength, UInt32>.Size) if totalLength > UInt32 (Tunnel.maximumMessageSize) {simpleTunnelLog ("Got a length that is too big: (totalLength) / self.closeTunnelWithError (SimpleTunnelError.internalError") as NSError totalLength UInt32 (return)} - = MemoryLayout< UInt32>.Size) read the packet payload. / Second, targetConnection.readMi NimumLength (Int (totalLength), maximumLength: Int (totalLength)) {data, error in if let payloadReadError {simpleTunnelLog = error ("Got an error on the tunnel connection: (payloadReadError) / self.closeTunnelWithError (payloadReadError") as NSError? Return let payloadData}) = Data Guard payloadData!.count (totalLength) {else = = Int simpleTunnelLog ("Payload data (length / (payloadData!.count))! = payload (length / (totalLength) self.closeTunnelWithError (SimpleTunnelError.internalError") as NSError) return _} = self.handlePacket (payloadData!) (self.readNextPacket) }

Call targetConnection.readMinimumLength to do a TCP connection monitoring, the monitoring server data. This method is called to read how much data can be set once, here we set MemoryLayout< UInt32>.Size, which is 4 bytes. So, because this project himself wrote a protocol.

2.3 custom transport protocol

Through the analysis of Tunnel and Tunnel.serializeMessage (Tunnel.sendMessage) and () and other methods can be found in every packet of Packet, the first 4 bytes, which is the length of the UInt32 position is stored throughout the length of the packet, the packet is behind the real content of Content, while Content content and Dictionary to store the different orders and content. A Pakcet structure is as follows:

4 byte totalLength n byte content data |

And this part of data parsing out is a Dictionary, where the Key is enum TunnelMessageKey., two of which in the send request data is inevitable

Key Value
Identifier The new TCP connection is generated
Command The current package command. See enum TunnelCommand.

2.4 monitor and parse packets

When data is received, enter the block of code in targetConnection.readMinimumLength

Determine if the current data is 4 bytes long. If this is not so long, the connection is out of order and you are reporting a false disconnect

Assign the total length of the package to the totalLength variable and subtract the 4 byte size of the tag bit that has been read

The rest of this part of the totalLength length is where we store the real data in the second half of our entire packet, and then call targetConnection.readMinimumLength to read the real data content

You can see in the code block reads data from the second layer of the first layer, like first determine the length, and then call self.handlePacket (payloadData!) to handle data packets, and then call the readNextPacket (recursive) monitor reads complete the next packet.

2.5 processing the contents of the packet payload

The most critical step in the 2.2 code is in self.handlePacket (payloadData), so let’s see what’s going on here. Let’s cut it a little bit

Process a message payload. func / / / handlePacket (packetData: Data _) -> Bool properties: [String: AnyObject] do {let {properties = try PropertyListSerialization.propertyList (from: packetData, options: PropertyListSerialization.MutabilityOptions), format: (NIL) as! [String: AnyObject] catch} {simpleTunnelLog ("Failed to create the message properties from the packet return false switch)}... CommandType.Data: guard let {case data = properties[TunnelMessageKey.Data.rawValue] as? Data else {break} / * check if the message has properties for host and port let host = properties[TunnelMessageKey.Host.rawValue] as / if String?, Let port = properties[TunnelMessageKey.Port.rawValue] as? Int {simpleTunnelLog ("Received data for connection / (connection?.identifier) / from (host) / (port): UDP case / *"): send peer's address along with data / targetConnection.sendDataWithEndPoint (data, host: host, port: port)} else {targetConnection.sendData} break (data) case.Suspend:... Case.Resume:... Case.Close: case.Packets: if let... Packets properties[TunnelMessageKey.Packets.rawValue] as = [Data], let = properties[TunnelMessageKey.Pro protocols? Tocols.rawValue] as? [NSNumber], packets.count = = protocols.count {targetConnection.sendPackets (packets protocols:, protocols break return (handleMessage)} default: commandType, properties: properties, connection: connection return true})}

First, the data is serialized, and then the order of the current package is determined

  • Data: here is the data for which endpoint server to specify. Use it in AppProxy. We don’t discuss it
  • Packets: is the system to send network requests, such as access to a web site, the package will be packaged into the data, sent to the endpoint server for parsing forward
  • Default: here is handling other requests. You need to process.OpenResult and.FetchConfiguration in the client. The details of its request are in ClientTunnel
Handle a message received from / / / the tunnel server. override func handleMessage (commandType: TunnelCommand properties: _, [String: AnyObject], connection: Connection? -> Bool) {var success = true switch commandType case.OpenResult: A logical {/ / connection was opened successfully. guard let targetConnection = connection, let = resultCodeNumber properties[TunnelMessageKey.ResultCode.rawValue] as? Int, let (resultCode = TunnelConnectionOpenResult rawValue: resultCodeNumber else = false break) {success} (targetConnection.handleOpenCompleted resultCode, properties:properties as [N SObject: AnyObject] case.FetchConfiguration: guard) let configuration = properties[TunnelMessageKey.Configuration.rawValue] as [String:? AnyObject] else {break} delegate?.tunnelDidSendConfiguration (self, configuration: configuration) default: simpleTunnelLog ("Tunnel received an invalid command success return success = false)}}

For the client, a server is connected processing the established data packet is complete authentication. Another is in the server network configuration content configuration of the current channel network, such as DNS, IP etc. in VPN, this is the most critical step.

In this DEMO, the new VPN connection channel, first authentication (Demo omitted), then the server configuration information. The correct set of configuration information, VPN is really set up. In this step the demo configuration on the PacketTunnelProvider.tunnelConnectionDidOpen.

Handle the event of the / / / logical flow of packets being established through the tunnel. func tunnelConnectionDidOpen (connection: ClientTunnelConnection configuration: [NSObject: AnyObject] _, the virtual) {/ / Create interface settings. guard let settings = createTunnelSettingsFromConfiguration (configuration) else (SimpleTunnelError.internalError as NSError pendingStartCompletion {?}) = pendingStartCompletion nil return the virtual interface settings. setTunnelNetworkSettings / Set (settings error in var startError: NSError) {let error = if? Error {simpleTunnelLog ("Failed to set the tunnel network settings: (error) /" startError = SimpleTunnelError.ba) DConfiguration as NSError} else {/ / Now we can start reading and writing packets to/from the virtual interface. self.tunnelConnection?.startHandlingPackets (the tunnel is)} / / Now fully established, call the start completion handler. self.pendingStartCompletion? (startError) self.pendingStartCompletion = nil}}

This step does just one thing: pass configuration information to the system, that is, calling the setTunnelNetworkSettings method to complete the process of building the VPN

NEPacketTunnelNetworkSettings preparation methods are in the createTunnelSettingsFromConfiguration method, here is not posted out, which is written in very intuitive

3. communicate normally with the server

Above we analysis basically is the process of establishing TCP connection to the network configuration information. The establishment of the TCP, the server automatically returns configuration information. Then at the request of the user, how to send and receive data?

IP packets with matching destination addresses will then be diverted to Packet Tunnel Provider and can be read using the packetFlow property. The Packet Tunnel Provider can then encapsulate the IP packets per a custom tunneling protocol and send them to a tunnel server. When the Packet Tunnel Provider decapsulates IP packets received from the tunnel server, it can use the packetFlow property to inject the packets into the networking stack.

The official has said that through packetFlow to send and receive systems, /app communications packets with the server

In theory, as long as the establishment of VPN to monitor round-trip packet. Look at the last piece of code, Section 2.5 of the self.tunnelConnection (.StartHandlingPackets)?, from the context and its name, should start listening to? We look at this method:

Make the initial readPacketsWithCompletionHandler call. / / / func (startHandlingPackets) {packetFlow.readPackets {inPackets, inProtocols in self.handlePackets (inPackets protocols: inProtocols)}}

Sure enough to monitor the data packets It is as expected, here NEPacketTunnelFlow.readPackets. According to the official statement, is to monitor virtual NIC TUN to PacketTunnelProvider data. The writePackets is the reverse operation. We look at what handlePackets had done.

Handle packets coming from the / / / packet flow. func handlePackets (_ packets: [Data], protocols: [NSNumber]) {guard let clientTunnel = tunnel as? ClientTunnel else {return} let properties = createMessagePropertiesForConnection (identifier, commandType:.Packets, extraProperties:[TunnelMessageKey.Packets.rawValue: packets as AnyObject, TunnelMessageKey.Protocols.rawValue: protocols as AnyObject) clientTunnel.sendMessage (properties) in if let sendError = {error error {self.delegate.tunnelConnectionDidClose (self, error: sendError as NSError? Return)} / / Read more packets. self.packetFlow.readPackets {inPackets, inProtocols in Self.handlePackets (inPackets, protocols:, inProtocols)}

Through the analysis of the previous sections, it should be easy to understand the contents of this. TUN sent the data package into the agreement, through the establishment of the TCP Tunnel to the server, and then re entered into the listening state, so the cycle monitoring.

Now that the data is sent to the server, where can I receive the data returned by the server?

Look back at the 2.2 section, readNextPacket () inside the logic to the last step is recursively self circulation monitoring server sends the content. While the 2.2 section and the section are calling a method called handlePackets, but two local calls are not the same method in this section is called. Read from the content of TUN, while the 2.2 section is from the server back to the packet. back to look at the 2.5 day treatment the contents of payload,.Packets which is processing the data package content. TargetConnection.sendPackets (packets, protocols: protocols) this is the key.

For the client, this connection should be the ClientTunnelConnection class. Go in and find out:

Send packets to the virtual / / / interface to be injected into the IP stack. override func sendPackets (_ packets: [Data], protocols: [NSNumber] packetFlow.writePackets (packets, withProtocols:) {protocols})

A simple step. Send the data back to TUN using the packetFlow.writePackets (packets, withProtocols:, protocols) method, and let the corresponding APP receive the data

4. summary

Here, the entire VPN, or TCP connection, is built into the logical source code that passes the data. After that, a sequence diagram is used to briefly sum up the entire VPN setup to the transport process:

Handle NetworkExtension: 2. analysis of the official Demo source code NEPacketTunnelProvider use part
VPN establishment process sequence diagram
Handle NetworkExtension: 2. analysis of the official Demo source code NEPacketTunnelProvider use part
transmission process sequence diagram

So the next step is based on the official source code for this analysis, and we’re going to code a simple VPN server and communication process. It’s still based on the ProxyDump project in the previous article

This is the end of this article

Please let me know and indicate the source, thank you