Routers are layer 3 devices because they transport packets at layer 3 (mostly IP). A switch is considered a layer 2 devices because it transport frames at layer 2 (mostly Ethernet). This means they respectively must-read information in the layer 3/2 header to be able to forward packets/frames.  Network routers only act on the network-layer fields of the layer-3 PDUs; they do not act on the transport-layer fields.

However, both switches and routers understand protocols above their respective transport layers. As an example, you can SSH to these devices. Does that make them work at the Application Layer? No, because in that case, they don't transport packet but are the source or destination. This is the same with RIP, OSPF, etc. These protocols are used by a router to forward packets.

Routing protocols, according to the OSI routing framework, are layer management protocols for the network layer, regardless of their transport mechanism: 

  • Open Shortest Path First (OSPF) is encapsulated in IP, but runs only on the IPv4 subnet, while the IPv6 version runs on the link using only link-local addressing.
  • Routing Information Protocol (RIP) runs over the User Datagram Protocol (UDP). Version 1 operates in broadcast mode, while version 2 uses multicast addressing.

Probing into RIP and OSPF packets using NetSim:

Following is a network scenario modeled in NetSim, which consists of two hosts exchanging CBR traffic. The Routers connecting them exchange routing information during the simulation. Routers support RIP and OSPF protocols.

Routing Information protocol (RIP):

Devices that use RIP will periodically (approximately every 30 s) broadcast routing information to neighboring hosts. The information sent by a host describes the devices they are directly connected to and the cost. RIP is not very scalable so is primarily used for small networks. RIP uses UDP to broadcast messages; port 520 is used by servers, whereas clients use a port above 1023. 

NetSim allows Wireshark packet capture in the routers which can be optionally enabled. Following is a Wireshark capture  done in the interface of Router 1 that is connected to Router 2 which contains RIP responses that were exchanged:

Following window displays the route information present in an RIP response message that Router 1 sends to Router 2:

You can also see that RIP messages use UDP port 520.

Open Shortest Path First (OSPF):

OSPF converges faster than RIP and it incorporates authentication. Interestingly, OSPF is not built on the transport layer but instead talks directly to IP. It is considered protocol 89 by the IP layer. OSPF messages are broadcast using two special multicast IP addresses: (all SPF/link state routers) and (all designated routers). The use of multicast addresses and setting the packet Time to Live (TTL) to one (which is done by OSPF) typically means a firewall will not pass this routing information.

Following is a Wireshark capture done in the interface of Router 1 that is connected to Router 2 that contains OSPF Hello Packets that were exchanged:

Following window displays the route information present in an OSPF Hello packet that Router 1 has multicasted:

You can also see that OSPF uses multicast IP sets the IP protocol filed to 89 as shown below: