[Route bulk] Fix bugs in case a SET operation follows a DEL operation in the same bulk (#2086)

**What I did**

Fix 2 issues in route bulk operation in the scenario where a SET operation follows a DEL operation on the same prefix.
1. The route entry SET operation can be ignored if it follows a DEL operation with the same prefix and next-hop.
2. DEL and then SET default route results in orchagent aborted

Added test case to cover the scenarios

**How I verified it**

Manually test
Unit test

**Details if related**
***The route entry SET operation can be ignored if it follows a DEL operation with the same prefix and next-hop.***
For example, the following operations
```
2021-12-18.06:17:54.793234|ROUTE_TABLE:1.1.1.0/24|SET|nexthop:10.0.0.33|ifname:Ethernet64
2021-12-18.06:19:53.187294|ROUTE_TABLE:1.1.1.0/24|DEL
2021-12-18.06:20:10.279850|ROUTE_TABLE:1.1.1.0/24|SET|nexthop:10.0.0.33|ifname:Ethernet64
```
result in the next sai operations
```
2021-12-18.06:19:31.203092|C|SAI_OBJECT_TYPE_ROUTE_ENTRY||{"dest":"1.1.1.0/24","switch_id":"oid:0x21000000000000","vr":"oid:0x3000000000010"}|SAI_ROUTE_ENTRY_ATTR_NEXT_HOP_ID=oid:0x4000000000aae
2021-12-18.06:20:23.813475|R|SAI_OBJECT_TYPE_ROUTE_ENTRY||{"dest":"1.1.1.0/24","switch_id":"oid:0x21000000000000","vr":"oid:0x3000000000010"}
```
and the last SET is lost.

This is because:
Both DEL and the 2nd SET operations were bundled into bulk operations.
There is a logic that is not to program ASIC if the next-hop information in the route entry update isn't changed,
which is done by comparing the next-hop information between route entry update and m_syncdRoutes.
However, in bulk mode, a route entry that will be removed will not be removed from m_syncdRoutes until the bulk is flushed,
The bulk has not been flushed when the 2nd SET is handled,
which means the comparison returns "not changed" and the 2nd SET operation is ignored.

The fix is to check whether the prefix is in gRouteBulker.removing_entries and treat it as non-exist if yes.

***DEL and then SET default route results in orchagent aborted***

This is because:
1. packet forward action is initialized when route entry is created
2. the default route will be set to DROP if the application asks for removing it
3. if the default route is readded later in the same bulk session,
   it doesn't update packet forward action but provides a valid next-hop
4. SAI complains if both DROP and a valid next hop are provided

Solution:
Always set packet action for the default route

Author: stephenxs

orchagent/routeorch.cpp

@@ -790,6 +790,11 @@ void RouteOrch::doTask(Consumer& consumer)
                     }
                 }
 
+                sai_route_entry_t route_entry;
+                route_entry.vr_id = vrf_id;
+                route_entry.switch_id = gSwitchId;
+                copy(route_entry.destination, ip_prefix);
+
                 if (nhg.getSize() == 1 && nhg.hasIntfNextHop())
                 {
                     if (alsv[0] == "unknown")
@@ -833,6 +838,7 @@ void RouteOrch::doTask(Consumer& consumer)
                 else if (m_syncdRoutes.find(vrf_id) == m_syncdRoutes.end() ||
                     m_syncdRoutes.at(vrf_id).find(ip_prefix) == m_syncdRoutes.at(vrf_id).end() ||
                     m_syncdRoutes.at(vrf_id).at(ip_prefix) != RouteNhg(nhg, ctx.nhg_index) ||
+                    gRouteBulker.bulk_entry_pending_removal(route_entry) ||
                     ctx.using_temp_nhg)
                 {
                     if (addRoute(ctx, nhg))
@@ -1874,6 +1880,25 @@ bool RouteOrch::addRoute(RouteBulkContext& ctx, const NextHopGroupKey &nextHops)
         }
         else
         {
+            if (!blackhole && vrf_id == gVirtualRouterId && ipPrefix.isDefaultRoute())
+            {
+                // Always set packet action for default route to avoid conflict settings
+                // in case a SET follows a DEL on the default route in the same bulk.
+                // - On DEL default route, the packet action will be set to DROP
+                // - On SET default route, as the default route has NOT been removed from m_syncdRoute
+                //   it calls SAI set_route_attributes instead of crate_route
+                //   However, packet action is called only when a route entry is created
+                //   This leads to conflict settings:
+                //   - packet action: DROP
+                //   - next hop: a valid next hop id
+                // To avoid this, we always set packet action for default route.
+                route_attr.id = SAI_ROUTE_ENTRY_ATTR_PACKET_ACTION;
+                route_attr.value.s32 = SAI_PACKET_ACTION_FORWARD;
+
+                object_statuses.emplace_back();
+                gRouteBulker.set_entry_attribute(&object_statuses.back(), &route_entry, &route_attr);
+            }
+
             route_attr.id = SAI_ROUTE_ENTRY_ATTR_NEXT_HOP_ID;
             route_attr.value.oid = next_hop_id;
 

tests/mock_tests/Makefile.am

@@ -23,6 +23,7 @@ LDADD_GTEST = -L/usr/src/gtest
 
 tests_SOURCES = aclorch_ut.cpp \
                 portsorch_ut.cpp \
+                routeorch_ut.cpp \
                 saispy_ut.cpp \
                 consumer_ut.cpp \
                 ut_saihelper.cpp \

tests/mock_tests/mock_orchagent_main.h

@@ -16,6 +16,9 @@
 #include "policerorch.h"
 #include "fgnhgorch.h"
 #include "flexcounterorch.h"
+#include "tunneldecaporch.h"
+#include "muxorch.h"
+#include "nhgorch.h"
 #include "directory.h"
 
 extern int gBatchSize;
@@ -44,6 +47,8 @@ extern FdbOrch *gFdbOrch;
 extern MirrorOrch *gMirrorOrch;
 extern BufferOrch *gBufferOrch;
 extern VRFOrch *gVrfOrch;
+extern NhgOrch *gNhgOrch;
+extern Srv6Orch  *gSrv6Orch;
 extern Directory<Orch*> gDirectory;
 
 extern sai_acl_api_t *sai_acl_api;
@@ -62,3 +67,4 @@ extern sai_hostif_api_t *sai_hostif_api;
 extern sai_buffer_api_t *sai_buffer_api;
 extern sai_queue_api_t *sai_queue_api;
 extern sai_udf_api_t* sai_udf_api;
+extern sai_mpls_api_t* sai_mpls_api;