feat: add Support for High-FPS Readings in Flashlight for Devices Over 60 FPS (#316)

* feat: read refresh rate specs from adb shell

* refactor(profiler): use a concise algorithm for extracting fps details

Use regular expression to find all matches of fps details from the adb command response. A contrived algorithm was used previously, relying on different regex constructs to isolate sections of the adb response strings such as strings that had the following format - "refreshRate %d", "fps=%d". The "refreshRate" string was dropped for the more common "fps".

chore(profiler): update base types with new method

Adding detectDeviceRefreshRate method to the base and sub classes. An implementation templation that can be adopted across multiple platforms to provide GPU refresh rate of the devices being profiled

* feat(webapp): send refresh rate to webapp from profiler by emiting an event in a socket

* chore(web-reporter): pass refresh rate numbers to reporter ui

chore: use device specs type

A new device spec has been created for use to model device spec relevant to performance

test: update snapshots and test cases

* refactor: extract socket events into an enum

* test: add unit test for verifying device refresh rate retrieval from ADB command

refactor: remove unnecessary catch block

* fix(profiler): accomodate varying refresh rate configurations

Some devices come with the capabilities of switching between 60,90, 120 fps. The renderFrameRate reports the currently configured refresh rate.

* test: add new specs for testing refresh rates on Pixels

* refactor(results): Add refresh rate to results model

For consistency, refresh rate is added to the DeviceSpecs property of the TestCaseResult and AveragedTestCaseResult

chore(test): update snapshots

refactor: remove unneeded changes

refactor: use logger for error

* refactor(sockets): place device refresh rates calls in start events

Author: MalcolmTomisin

packages/commands/measure/src/__tests__/__snapshots__/measure.test.tsx.snap

@@ -12,15 +12,16 @@ Time taken to run the test.
 Can be helpful to measure Time To Interactive of your app, if the test is checking app start for instance.
 Average FPS
 60 FPS
-Frame Per Second. Your app should display 60 Frames Per Second to give an impression of fluidity. This number should be close to 60, otherwise it will seem laggy. 
+Frame Per Second. Your app should display 60 Frames Per Second to give an impression of fluidity. This number should be close to 60, otherwise it will seem laggy.
+ 
 See
  
 this video
  
 for more details
 Average CPU usage
 83 %
-An app might run at 60FPS but might be using too much processing power, so it's important to check CPU usage.
+An app might run at high frame rates, such as 60 FPS or higher, but might be using too much processing power, so it's important to check CPU usage.
  Depending on the device, this value can go up to 
 100% x number of cores
 . For instance, a Samsung A10s has 4 cores, so the max value would be 400%.
@@ -855,7 +856,7 @@ exports[`flashlight measure interactive it displays measures: Web app with measu
                     <div
                       class="text-neutral-400 text-sm"
                     >
-                      An app might run at 60FPS but might be using too much processing power, so it's important to check CPU usage.
+                      An app might run at high frame rates, such as 60 FPS or higher, but might be using too much processing power, so it's important to check CPU usage.
                       <br />
                        Depending on the device, this value can go up to 
                       <code>
@@ -3812,15 +3813,16 @@ Time taken to run the test.
 Can be helpful to measure Time To Interactive of your app, if the test is checking app start for instance.
 Average FPS
 -
-Frame Per Second. Your app should display 60 Frames Per Second to give an impression of fluidity. This number should be close to 60, otherwise it will seem laggy. 
+Frame Per Second. Your app should display 60 Frames Per Second to give an impression of fluidity. This number should be close to 60, otherwise it will seem laggy.
+ 
 See
  
 this video
  
 for more details
 Average CPU usage
 -
-An app might run at 60FPS but might be using too much processing power, so it's important to check CPU usage.
+An app might run at high frame rates, such as 60 FPS or higher, but might be using too much processing power, so it's important to check CPU usage.
  Depending on the device, this value can go up to 
 100% x number of cores
 . For instance, a Samsung A10s has 4 cores, so the max value would be 400%.
@@ -4218,7 +4220,7 @@ exports[`flashlight measure interactive it displays measures: Web app with no me
                     <div
                       class="text-neutral-400 text-sm"
                     >
-                      An app might run at 60FPS but might be using too much processing power, so it's important to check CPU usage.
+                      An app might run at high frame rates, such as 60 FPS or higher, but might be using too much processing power, so it's important to check CPU usage.
                       <br />
                        Depending on the device, this value can go up to 
                       <code>

packages/commands/measure/src/server/ServerSocketConnectionApp.tsx

@@ -1,9 +1,10 @@
 import { PerformanceMeasurer } from "@perf-profiler/e2e";
 import { Logger } from "@perf-profiler/logger";
+import { profiler } from "@perf-profiler/profiler";
 import { Measure } from "@perf-profiler/types";
 import React, { useCallback, useEffect } from "react";
 import { HostAndPortInfo } from "./components/HostAndPortInfo";
-import { SocketType } from "./socket/socketInterface";
+import { SocketType, SocketEvents } from "./socket/socketInterface";
 import { useSocketState, updateMeasuresReducer, addNewResultReducer } from "./socket/socketState";
 import { useBundleIdControls } from "./useBundleIdControls";
 import { useLogSocketEvents } from "../common/useLogSocketEvents";
@@ -33,9 +34,11 @@ export const ServerSocketConnectionApp = ({ socket, url }: { socket: SocketType;
         )
       );
 
-    socket.on("start", async () => {
+    socket.on(SocketEvents.START, async () => {
+      const refreshRate = profiler.detectDeviceRefreshRate();
       setState({
         isMeasuring: true,
+        refreshRate,
       });
 
       if (!state.bundleId) {
@@ -55,19 +58,19 @@ export const ServerSocketConnectionApp = ({ socket, url }: { socket: SocketType;
       );
     });
 
-    socket.on("stop", stop);
+    socket.on(SocketEvents.STOP, stop);
 
-    socket.on("reset", () => {
+    socket.on(SocketEvents.RESET, () => {
       stop();
       setState({
         results: [],
       });
     });
 
     return () => {
-      socket.removeAllListeners("start");
-      socket.removeAllListeners("stop");
-      socket.removeAllListeners("reset");
+      socket.removeAllListeners(SocketEvents.START);
+      socket.removeAllListeners(SocketEvents.STOP);
+      socket.removeAllListeners(SocketEvents.RESET);
     };
   }, [setState, socket, state.bundleId, stop]);
 

packages/commands/measure/src/server/socket/socketInterface.ts

@@ -5,6 +5,7 @@ export interface SocketData {
   isMeasuring: boolean;
   bundleId: string | null;
   results: TestCaseResult[];
+  refreshRate: number;
 }
 
 export interface ServerToClientEvents {
@@ -18,6 +19,7 @@ export interface ClientToServerEvents {
   reset: () => void;
   autodetectBundleId: () => void;
   setBundleId: (bundleId: string) => void;
+  autodetectRefreshRate: () => void;
 }
 
 interface InterServerEvents {
@@ -37,3 +39,17 @@ export type SocketType = Socket<
   InterServerEvents,
   SocketData
 >;
+
+export enum SocketEvents {
+  START = "start",
+  STOP = "stop",
+  RESET = "reset",
+  AUTODETECT_BUNDLE_ID = "autodetectBundleId",
+  SET_BUNDLE_ID = "setBundleId",
+  AUTODETECT_REFRESH_RATE = "autodetectRefreshRate",
+  UPDATE_STATE = "updateState",
+  SEND_ERROR = "sendError",
+  PING = "ping",
+  CONNECT = "connect",
+  DISCONNECT = "disconnect",
+}

packages/commands/measure/src/server/socket/socketState.ts

@@ -1,12 +1,13 @@
 import { Measure, POLLING_INTERVAL } from "@perf-profiler/types";
 import { useState, useEffect } from "react";
-import { SocketType, SocketData } from "./socketInterface";
+import { SocketType, SocketData, SocketEvents } from "./socketInterface";
 
 export const useSocketState = (socket: SocketType) => {
   const [state, _setState] = useState<SocketData>({
     isMeasuring: false,
     bundleId: null,
     results: [],
+    refreshRate: 60,
   });
 
   const setState = (
@@ -23,7 +24,7 @@ export const useSocketState = (socket: SocketType) => {
   };
 
   useEffect(() => {
-    socket.emit("updateState", state);
+    socket.emit(SocketEvents.UPDATE_STATE, state);
   }, [state, socket]);
 
   return [state, setState] as const;
@@ -54,6 +55,9 @@ export const addNewResultReducer = (state: SocketData, name: string): SocketData
       name,
       iterations: [],
       status: "SUCCESS",
+      specs: {
+        refreshRate: state.refreshRate,
+      },
     },
   ],
 });

packages/commands/measure/src/server/useBundleIdControls.ts

@@ -1,20 +1,20 @@
 import { profiler } from "@perf-profiler/profiler";
 import { useEffect } from "react";
-import { SocketType, SocketData } from "./socket/socketInterface";
+import { SocketType, SocketData, SocketEvents } from "./socket/socketInterface";
 
 export const useBundleIdControls = (
   socket: SocketType,
   setState: (state: Partial<SocketData>) => void,
   stop: () => void
 ) => {
   useEffect(() => {
-    socket.on("setBundleId", (bundleId) => {
+    socket.on(SocketEvents.SET_BUNDLE_ID, (bundleId) => {
       setState({
         bundleId,
       });
     });
 
-    socket.on("autodetectBundleId", () => {
+    socket.on(SocketEvents.AUTODETECT_BUNDLE_ID, () => {
       stop();
 
       try {
@@ -23,13 +23,26 @@ export const useBundleIdControls = (
           bundleId,
         });
       } catch (error) {
-        socket.emit("sendError", error instanceof Error ? error.message : "unknown error");
+        socket.emit(
+          SocketEvents.SEND_ERROR,
+          error instanceof Error ? error.message : "unknown error"
+        );
       }
     });
 
+    socket.on(SocketEvents.AUTODETECT_REFRESH_RATE, () => {
+      stop();
+
+      const refreshRate = profiler.detectDeviceRefreshRate();
+      setState({
+        refreshRate,
+      });
+    });
+
     return () => {
-      socket.removeAllListeners("setBundleId");
-      socket.removeAllListeners("autodetectBundleId");
+      socket.removeAllListeners(SocketEvents.SET_BUNDLE_ID);
+      socket.removeAllListeners(SocketEvents.AUTODETECT_BUNDLE_ID);
+      socket.removeAllListeners(SocketEvents.AUTODETECT_REFRESH_RATE);
     };
   }, [setState, socket, stop]);
 };

packages/commands/measure/src/webapp/components/SocketState.tsx

@@ -8,6 +8,7 @@ import Button from "@mui/material/Button";
 import { Logger } from "@perf-profiler/logger";
 import { socket } from "../socket";
 import { useLogSocketEvents } from "../../common/useLogSocketEvents";
+import { SocketEvents } from "../../server/socket/socketInterface";
 
 const useSocketState = (onError: (error: string) => void) => {
   useLogSocketEvents(socket);
@@ -28,14 +29,14 @@ const useSocketState = (onError: (error: string) => void) => {
       }
     }
 
-    socket.on("connect", onConnect);
-    socket.on("disconnect", onDisconnect);
-    socket.on("sendError", onError);
+    socket.on(SocketEvents.CONNECT, onConnect);
+    socket.on(SocketEvents.DISCONNECT, onDisconnect);
+    socket.on(SocketEvents.SEND_ERROR, onError);
 
     return () => {
-      socket.off("connect", onConnect);
-      socket.off("disconnect", onDisconnect);
-      socket.off("sendError", onError);
+      socket.off(SocketEvents.CONNECT, onConnect);
+      socket.off(SocketEvents.DISCONNECT, onDisconnect);
+      socket.off(SocketEvents.SEND_ERROR, onError);
     };
   }, [onError]);
 

packages/commands/measure/src/webapp/socket.ts

@@ -1,8 +1,12 @@
 import { io, Socket } from "socket.io-client";
-import { ServerToClientEvents, ClientToServerEvents } from "../server/socket/socketInterface";
+import {
+  ServerToClientEvents,
+  ClientToServerEvents,
+  SocketEvents,
+} from "../server/socket/socketInterface";
 
 export const socket: Socket<ServerToClientEvents, ClientToServerEvents> = io(
   window.__FLASHLIGHT_DATA__.socketServerUrl
 );
 
-socket.on("disconnect", () => socket.close());
+socket.on(SocketEvents.DISCONNECT, () => socket.close());

packages/commands/measure/src/webapp/useMeasures.ts

@@ -1,36 +1,38 @@
 import { useEffect, useState } from "react";
-import type { SocketData } from "../server/socket/socketInterface";
+import { SocketData, SocketEvents } from "../server/socket/socketInterface";
 import { socket } from "./socket";
 
 export const useMeasures = () => {
   const [state, setState] = useState<SocketData>();
 
   useEffect(() => {
-    socket.on("updateState", setState);
+    socket.on(SocketEvents.UPDATE_STATE, setState);
 
     return () => {
-      socket.off("updateState", setState);
+      socket.off(SocketEvents.UPDATE_STATE, setState);
     };
   }, []);
 
   return {
     bundleId: state?.bundleId ?? null,
+    refreshRate: state?.refreshRate ?? 60,
     autodetect: () => {
-      socket.emit("autodetectBundleId");
+      socket.emit(SocketEvents.AUTODETECT_BUNDLE_ID);
+      socket.emit(SocketEvents.AUTODETECT_REFRESH_RATE);
     },
     setBundleId: (bundleId: string) => {
-      socket.emit("setBundleId", bundleId);
+      socket.emit(SocketEvents.SET_BUNDLE_ID, bundleId);
     },
     results: state?.results ?? [],
     isMeasuring: state?.isMeasuring ?? false,
     start: () => {
-      socket.emit("start");
+      socket.emit(SocketEvents.START);
     },
     stop: () => {
-      socket.emit("stop");
+      socket.emit(SocketEvents.STOP);
     },
     reset: () => {
-      socket.emit("reset");
+      socket.emit(SocketEvents.RESET);
     },
   };
 };

packages/core/reporter/src/reporting/Report.ts

@@ -117,4 +117,8 @@ export class Report {
       threads: getThreadsStats(iterations),
     };
   }
+
+  public getRefreshRate() {
+    return this.result.specs?.refreshRate ?? 60;
+  }
 }

packages/core/reporter/src/reporting/getScore.ts

@@ -26,7 +26,7 @@ export const getScore = (result: AveragedTestCaseResult) => {
   const scores = [cpuScore];
 
   if (averageUIFPS !== undefined) {
-    const fpsScore = (averageUIFPS * 100) / 60;
+    const fpsScore = (averageUIFPS * 100) / (result?.specs?.refreshRate ?? 60);
     scores.push(fpsScore);
   }
 

packages/core/types/index.ts

@@ -39,6 +39,7 @@ export interface TestCaseResult {
   status: TestCaseResultStatus;
   iterations: TestCaseIterationResult[];
   type?: TestCaseResultType;
+  specs?: DeviceSpecs;
 }
 
 export interface AveragedTestCaseResult {
@@ -49,6 +50,7 @@ export interface AveragedTestCaseResult {
   average: TestCaseIterationResult;
   averageHighCpuUsage: { [processName: string]: number };
   type?: TestCaseResultType;
+  specs?: DeviceSpecs;
 }
 
 // Shouldn't really be here but @perf-profiler/types is imported by everyone and doesn't contain any logic
@@ -97,4 +99,9 @@ export interface Profiler {
   cleanup: () => void;
   getScreenRecorder: (videoPath: string) => ScreenRecorder | undefined;
   stopApp: (bundleId: string) => Promise<void>;
+  detectDeviceRefreshRate: () => number;
+}
+
+export interface DeviceSpecs {
+  refreshRate: number;
 }