fix(Android, SAV+Tabs+Stack v4): fix CustomToolbar's insets handling to use insets received from ancestor views

[kligarski]

Description

Adapts CustomToolbar from Stack v4 to use insets received from ancestor views via onApplyWindowInsets.

Screen.Recording.2025-09-23.at.10.12.13.mov

Closes https://github.com/software-mansion/react-native-screens-labs/issues/435.

Reasoning

On API >= 30, changing rootWindowInsets to unhandledInsets (insets received from onApplyWindowInsets) is enough. Unfortunately, on API < 30, there is no insets section for display cutout, only cutout details are passed (.getDisplayCutout()) with possibility to set mDisplayCutoutConsumed flag. Setting setInsets(WindowInsetsCompat.Type.displayCutout(), Insets.NONE) does nothing on API < 30. When we read those insets, we get insets returned from .getDisplayCutout()'s safe area. If WindowInsetsCompat has been created with WindowInsetsCompat toWindowInsetsCompat(@NonNull WindowInsets insets, @Nullable View view), cutout insets from root view are used instead of insets passed from ancestor views (source):

case Type.DISPLAY_CUTOUT: {
    // Visibility does not affect this type of inset
    final DisplayCutoutCompat cutout = mRootWindowInsets != null
            ? mRootWindowInsets.getDisplayCutout()
            : getDisplayCutout();
    if (cutout != null) {
        return Insets.of(cutout.getSafeInsetLeft(), cutout.getSafeInsetTop(),
                cutout.getSafeInsetRight(), cutout.getSafeInsetBottom());
    } else {
        return Insets.NONE;
    }
}

We want to:

• consume displayCutout insets:
  • that's why we use older .consumeDisplayCutout() API to consume insets on SDK < 30
• use displayCutout insets from ancestor views, also on SDK < 30:
  • that's why I removed passing view to .toWindowInsetsCompat() in utils/InsetsKt.kt

I also removed insets ignoringVisibility, as we want to handle only visible bars.

Testing

(**) is described in more detail in the section below

Issue mentioned in CustomToolbar

// We use rootWindowInsets in lieu of insets or unhandledInsets here,
// because cutout sometimes (only in certain scenarios, e.g. with headerLeft view present)
// happen to be Insets.ZERO and is not reliable.

I did not manage to reproduce the issue mentioned in the comment above, even on older API versions.

Current problems

Nested header redundantly accounts for insets

(visible in the video below)

This will be fixed only if we decide to add support to SafeAreaView for Stack v4 on Android. This will require some additional logic (overriding dispatchApplyWindowInsets) to pass insets consumed by CustomToolbar to Screen's child.

Ticket for this: https://github.com/software-mansion/react-native-screens-labs/issues/464.

Oversized margin top for title in nested header

Screen_recording_20250923_112144.mp4

This is not a regression, it also happens on current main. It will require more investigation - ticket for this: https://github.com/software-mansion/react-native-screens-labs/issues/463.

Bug when dynamically changing safe area view edges

This is the specific case when I noticed this happening:

Screen.Recording.2025-09-23.at.10.15.45.mov

I haven't debugged this yet. Ticket for this: https://github.com/software-mansion/react-native-screens-labs/issues/462.

Insets behave odd when changing orientation between landscapeLeft/Right to landscapeRight/left

Screen.Recording.2025-09-22.at.14.15.40.mov

This is not a regression, ticket for this: https://github.com/software-mansion/react-native-screens-labs/issues/461.

Changes

• use insets from onApplyWindowInsets insetad of root view insets in CustomToolbar
• use regular .toWindowInsetsCompat that does not use root view to determine insets

Test code and steps to reproduce

Run TestBottomTabs with SafeAreaView in Tab4.

Checklist

• Included code example that can be used to test this change
• Updated TS types
• Updated documentation:
  • https://github.com/software-mansion/react-native-screens/blob/main/guides/GUIDE_FOR_LIBRARY_AUTHORS.md
  • https://github.com/software-mansion/react-native-screens/blob/main/native-stack/README.md
  • https://github.com/software-mansion/react-native-screens/blob/main/src/types.tsx
  • https://github.com/software-mansion/react-native-screens/blob/main/src/native-stack/types.tsx
• Ensured that CI passes

[kkafar]

I like the PR overall & appreciate the description.

I'll test the runtime soon.